Blog / Research notes
Why Models Drift
A shared perspective on drift, from LLM reinforcement learning to autoregressive video.
A reasoning model suddenly collapses during reinforcement learning. An autoregressive video looks less and less like its opening frames. A quantized KV cache lets historical frames “steal” attention from the current frame. These seem like separate problems: optimization instability, temporal consistency, and numerical precision. Yet they share a structure worth studying: the model helps construct the distribution it will encounter next, while the training or update rule does not always make correct assumptions about that distribution.
This article brings together Score Centering, Bellman Policy Optimization (BPO), and Maximum Likelihood Reinforcement Learning (MaxRL) with several approaches to video generation. The aim is not to give them all the same label, but to answer a more specific question: where does bias enter the feedback loop, and where should we intervene?
The central distinction: Score Centering corrects systematic bias in updates; Self Forcing changes the histories encountered during training; BPO rewrites the policy optimization objective; MaxRL changes the learning weight assigned to successful trajectories. These methods can complement one another, but they are not interchangeable.
Two kinds of drift, two clocks
First, we need to clarify what advances when a model “drifts further.”
In LLM reinforcement learning, the trainer produces parameters, an inference engine uses those parameters to generate rollouts, and the trainer updates the parameters from those rollouts. Quantization, different numerical kernels, asynchronous sampling, and stale checkpoints can all cause the inference engine and trainer to assign different probabilities to the same token. Once bias enters the update, the new parameters are sent back to the sampler, creating feedback across training steps.
AR video has another feedback chain: the model generates the next chunk from its history, adds that chunk to the history, and uses it to generate what follows. Even with completely fixed weights, a small error in a person’s shape or a slight shift in geometry can enter the conditioning history and continue to propagate.
LLM RL · Training step k
Parameters θk → sampler → rollouts → gradient → parameters θk+1
The evolving state includes model parameters; biased updates are repeatedly fed back.
AR video · Generation step t
History ht → next video chunk → append to history → ht+1
The parameters can remain fixed; what changes is the history the model reads next.
Both systems have feedback loops, but different things accumulate. In Score Centering, drift is a precisely defined term in a gradient decomposition. In video, drift usually refers to the degradation of identity, geometry, motion, or scene structure over time. Reducing the latter entirely to the former would obscure where changes are actually needed.
A useful local analytical perspective is:
\[e_{n+1}\approx J_n e_n+b_n+\xi_n.\]Here, \(e_n\) is a perturbation in a chosen representation, \(J_n\) describes how it propagates, \(b_n\) is systematic bias, and \(\xi_n\) is random disturbance. This is a local approximation for understanding mechanisms, not a drift theorem shared by all generative models. Bias does not necessarily make a system diverge; whether the propagation operator amplifies errors matters too.
The simple model below isolates this distinction. It has just one scalar state, starting at \(e_0=0.2\). The solid line follows \(e_{n+1}=a e_n+b\), while the dashed line removes only \(b\) and retains exactly the same propagation coefficient \(a\).
With the default parameters, the errors after 40 steps are 1.2 and 0.2, respectively.
This example explains why “correcting bias” and “improving robustness in a feedback loop” are different tasks. The former addresses the newly injected \(b_n\); the latter must also consider how the system continues operating when errors already exist.
Locating mismatch at different levels of the distribution
For an input \(x\), a generated history \(h_t\), and the next generated unit \(a_t\), we can write the actual sampling process as:
\[x\sim\rho,\qquad h_t\sim d_t^q(\cdot\mid x),\qquad a_t\sim q(\cdot\mid h_t).\]\(\rho\) is the prompt distribution, \(d_t^q\) is the distribution of histories visited by the sampler at step \(t\), and \(q\) is the conditional sampling distribution given a history. Let \(p_\theta\) denote the probabilities the trainer uses to compute gradients. “Mismatch between training and actual operation” can therefore occur in at least three places.
| Level | What should we ask? | Representative interventions |
|---|---|---|
| Conditional distribution at a fixed history | At the same \(h\), do the sampler’s \(q\) and the trainer’s \(p_\theta\) agree? | Score Centering; bias correction for computational operators |
| Distribution of visited histories | Does the training distribution \(d_t\) cover the histories the model actually generates? | Self Forcing; training on actual rollouts |
| Weights on inputs and successful trajectories | Which prompts and complete outputs receive more learning signal? | MaxRL; success-conditioned or reward-weighted fitting |
BPO asks another question that cuts across this table: which overall policy objective does the local loss actually approximate? Sometimes the difficulty comes from a distribution mismatch, sometimes from the design of a surrogate objective, and sometimes from both.
One easily confused point is that the sum of teacher-forced token NLLs is already the exact sequence NLL:
\[-\log p_\theta(a_{1:T}\mid x) =-\sum_t\log p_\theta(a_t\mid x,a_{<t}).\]So the problem is not that “token losses inherently cannot see a whole trajectory.” It is that, when a model has finite error, learning conditional behavior on real prefixes does not mean it can also recover from the off-distribution prefixes it generates itself. The video distribution-matching surrogate objectives discussed here also cannot simply be equated with maximum likelihood on real data.
Score Centering: removing an update that should not exist
Score Centering starts from a powerful test: if every output receives the same positive reward, should the model keep learning? Under standard on-policy policy gradients, the answer is no, because the environment supplies no signal distinguishing better outputs from worse ones.
Fix a history \(h\) and define the parameter score:
\[g_\theta(a,h)=\nabla_\theta\log p_\theta(a\mid h).\]Under the usual differentiability and normalization conditions, \(\mathbb E_{p_\theta}[g_\theta\mid h]=0\). But when tokens are actually sampled from \(q\), \(\mathbb E_q[g_\theta\mid h]\) can generally be nonzero. Taking the conditional expectation over the current action and its subsequent rollout, the update decomposes into:
\[\mathbb E_q[Rg_\theta\mid h] =\underbrace{\mathbb E_q[R\mid h]\, \mathbb E_q[g_\theta\mid h]}_{\text{drift}} +\underbrace{\operatorname{Cov}_q(R,g_\theta\mid h)}_{\text{reward-related signal}}.\]The first term depends on reward only through its mean; it does not care which action leads to a better outcome. For a positive conditional mean reward, it pushes the trainer toward the sampler. If the sampler is a fixed teacher, this resembles ordinary distillation. If it is a quantized or stale copy of the trainer, the trainer follows it each round and sends its updated parameters back, allowing bias to accumulate around the loop.
Score Centering directly replaces:
\[g_\theta(a,h)\longrightarrow g_\theta(a,h)-\mathbb E_q[g_\theta\mid h].\]With the full conditional expectation, the drift term is removed exactly, leaving \(\operatorname{Cov}_q(R,g_\theta\mid h)\). But this is still a covariance under \(q\), not a full restoration of the on-policy update under \(p_\theta\). The paper’s practical implementation further approximates this expectation using the sampler’s top-\(k\) probabilities and the trainer’s tail distribution.
This also explains why centering rewards or advantages within a prompt is insufficient. Having advantages sum to zero across a whole rollout group does not mean the conditional mean advantage is zero at every intermediate prefix. A prefix already close to a correct answer and a prefix that has already gone wrong generally have different conditional means.
Video counterpart: attention bias after KV quantization
In Quantized Keys Steal Attention, historical keys in AR video are compressed to low precision, while keys in the current chunk retain full precision. Even if quantization error is approximately zero-mean in attention logits, the exponential inside softmax changes the mean:
\[\hat\ell_i=\ell_i+\delta_i,\quad \mathbb E[\delta_i]=0, \qquad \mathbb E[e^{\hat\ell_i}] =e^{\ell_i}\mathbb E[e^{\delta_i}] \ge e^{\ell_i}.\]The unnormalized contribution of historical keys is systematically inflated, shifting attention toward cached tokens. The paper derives an additive correction for each attention score:
\[b_i=\log\mathbb E[e^{\delta_i}],\qquad \tilde\ell_i=\hat\ell_i-b_i, \qquad b_i\approx\tfrac12\operatorname{Var}(\delta_i).\]When the noise model holds and the exact \(b_i\) is used, \(\mathbb E[e^{\tilde\ell_i}]=e^{\ell_i}\). The practical method uses a second-order approximation and requires no retraining. The paper validates it on MAGI-1, SkyReels-V2, and HY-WorldPlay.
The closest connection to Score Centering is this causal path: efficient computation introduces bias → analyze the expected bias → compensate with an additive term, without requiring efficient computation to be abandoned. Here, however, the restored quantity is the expectation of the unnormalized attention contribution. This does not directly imply that normalized softmax is fully unbiased, much less that arbitrarily long videos will no longer drift.
Three meanings of “score” are easy to confuse here: in RL, the parameter score is ∇θ log p; an attention score is a logit before softmax; in diffusion models, the score is usually ∇x log p. Sharing a name does not make their formulas interchangeable.
BPO: deriving a local loss from trajectory relations
BPO begins with Policy Mirror Descent (PMD), rather than replacing a coefficient in an existing GRPO loss. For a fixed rollout policy \(\mu\), PMD balances advantage against the KL cost of deviating from the old policy at each state:
\[\max_\pi\; \mathbb E_{a\sim\pi(\cdot\mid h)}[A^\mu(h,a)] -\frac{1}{\eta}D_{\mathrm{KL}}(\pi\Vert\mu).\]Its ideal solution satisfies \(\pi^+(a\mid h)\propto\mu(a\mid h)e^{\eta A^\mu(h,a)}\). The difficulty is that directly using the advantage of every prefix in language generation usually requires estimating values for many intermediate states.
BPO uses the fact that appending a token is deterministic and that reward is given only at the endpoint. Following the paper’s convention, write the terminal reward as a terminal value, \(V^\mu(h_{T+1})=R\). Then:
\[A^\mu(h_t,a_t)=V^\mu(h_{t+1})-V^\mu(h_t), \qquad \sum_tA^\mu(h_t,a_t)=R-V^\mu(x).\]The intermediate values cancel along the path, leaving only the terminal reward and the expected reward of the initial prompt. The latter still needs to be estimated, but grouped rollouts for the same prompt can do this without separately training a critic for intermediate states.
A direct connection between BPO and Score Centering
Define the per-token term in BPO’s full objective:
\[u_\theta(a,h)= \log\frac{p_\theta(a\mid h)}{\mu(a\mid h)} +D_{\mathrm{KL}}\!\left(\mu(\cdot\mid h)\Vert p_\theta(\cdot\mid h)\right).\]When \(p_\theta=\pi^+\), PMD’s optimality condition gives the trajectory relation \(\sum_tu_\theta=\eta(R-V^\mu(x))\). BPO therefore constructs a squared-residual objective:
\[\delta=\eta(R-V^\mu(x))-\sum_tu_\theta(a_t,h_t), \qquad L=\mathbb E_{x\sim\rho,\,\tau\sim P_\mu(\cdot\mid x)} \left[\frac{\phi(x)}{2\eta}\delta^2\right],\quad\phi(x)>0.\]Within the policy space and reachable states assumed in the paper, the full objective has the same optimum as PMD. This is a statement about the objective, not a guarantee that any neural network and optimizer will reach that solution.
The equations above also yield an interesting connection directly. Differentiating while holding \(\mu\) fixed:
\[\nabla_\theta u_\theta(a,h) =g_\theta(a,h)-\mathbb E_{b\sim\mu}[g_\theta(b,h)].\]The right-hand side is exactly score centering with respect to the actual sampling distribution. This is an algebraic observation obtained from the equations in the two papers: the full KL term that BPO derives from an overall policy objective contains SC’s centering structure in its gradient. It does not mean the two algorithms have identical objectives and updates in every respect.
Practical BPO additionally applies residual linearization, group estimation, a binary KL approximation, smoothing, and clipping, ultimately using a complementary-probability weight:
\[\omega_t= \frac{1+\epsilon-\mu(a_t\mid h_t)} {1+\epsilon-p_\theta(a_t\mid h_t)}.\]The practical token loss therefore does not automatically inherit the exact zero-mean property of full-KL centering. This step is a particularly useful reminder: knowing the final loss is not enough; we must also know which conditions were discarded on the way from the overall objective.
Video counterpart: analytical elimination in Flow-DPO
Flow-DPO in Improving Video Generation with Human Feedback likewise starts from an objective over the complete generation distribution. For a video \(v\) and condition \(c\), the ideal solution to KL-regularized reward maximization has the form:
\[p^*(v\mid c)=\frac{p_{\rm ref}(v\mid c)e^{R(v,c)/\beta}}{Z(c)}.\]Rearranging this relation expresses reward in terms of a ratio of generation probabilities. For preferred and rejected videos under the same prompt, the reward difference cancels \(\log Z(c)\), yielding a preference optimization formulation that does not require an explicit reward model in that training objective. A diffusion/flow likelihood surrogate then turns this into training on denoising or velocity prediction errors.
The methodological parallel is clear:
BPO
PMD → Bellman elimination of intermediate values → trajectory residual → token loss
Flow-DPO
KL-regularized reward objective → eliminate reward and normalizer → video preferences → velocity loss
This is a correspondence between derivation patterns. Flow-DPO does not use BPO’s Bellman elimination or its complementary-probability weights. Practical Flow-DPO also replaces the time-dependent \(\beta_t\) from its derivation with a constant to improve training. Here, time means denoising time, not the video frame index.
To actually transfer BPO to video, one cannot simply copy \((1-\mu)/(1-p)\): it comes from binary KL on discrete token probabilities. The probability density of a continuous latent can exceed 1, so “one minus the density” is no longer the probability of a complementary event. A new derivation is needed for the specific video states, actions, and sampler.
MaxRL: how successful samples should be counted
MaxRL addresses another kind of mismatch: average success probability and the log-likelihood of success are different objectives. Let \(s_\theta(x)\) be the success probability for prompt \(x\). Then:
\[J_{\rm RL}=\mathbb E_{x\sim\rho}[s_\theta(x)], \qquad J_{\rm ML}=\mathbb E_{x\sim\rho}[\log s_\theta(x)].\]Both encourage success, but allocate gradients differently. The latter weights by \(1/s_\theta(x)\): in the gradient expression, problems with rare successes receive greater relative weight. This changes the allocation of learning signal; it does not correct probability differences between sampler and trainer.
Let \(\tau\) be a complete trajectory and let success be determined by a binary criterion. Under the required differentiability and support conditions:
\[\nabla_\theta\log s_\theta(x) =\mathbb E_{\tau\sim P_\theta(\cdot\mid x,\mathrm{success})} [\nabla_\theta\log P_\theta(\tau\mid x)].\]This gives maximum likelihood a simple interpretation: average the scores of successful trajectories. For a prompt, draw independent samples from the current \(P_\theta(\cdot\mid x)\): \(N\) trajectories in total, of which \(K\) succeed. We can construct an estimator that averages only successful trajectories:
\[\hat g_N(x)= \begin{cases} \frac{1}{K}\sum_{i=1}^{N}r_i\nabla_\theta\log P_\theta(\tau_i\mid x),&K>0,\\ 0,&K=0. \end{cases}\]With finite sampling, success will not be observed for every prompt, so this is not an unbiased estimator of the exact ML gradient. MaxRL’s key result is that its expectation corresponds exactly to a truncated objective:
\[J_N(x)=-\sum_{j=1}^{N}\frac{(1-s_\theta(x))^j}{j}, \qquad \nabla J_N(x)=\sum_{j=1}^{N}\frac1j\nabla\operatorname{pass@}j(x).\]At \(N=1\), this reduces to the standard expected-reward objective, up to a constant; as \(N\to\infty\), it approaches log-likelihood. Increasing the rollout count therefore does more than reduce estimation variance for a fixed objective: it changes the objective being approximated. This discussion concerns the estimator that averages only successes; practical implementations with a baseline must be analyzed according to their specific form.
Video counterpart: LiFT’s success-filtering variant
The main method in LiFT is reward-weighted learning: generated videos are scored by a learned evaluation model, higher-reward videos receive larger training weights, and a training loss on real videos is mixed in. It also studies a rejection sampling variant: keep only videos rated Good on all three dimensions—semantic consistency, motion smoothness, and video fidelity—and fine-tune on those samples.
“Fitting the generation distribution conditioned on success” is close to MaxRL, but the way samples are normalized changes the result. Suppose two prompts each produce 10 videos: A has 8 successes and B has 1.
| How are these 9 successful videos used? | Prompt A | Prompt B |
|---|---|---|
| Pool all successes and weight every video equally | Contributes 8 training samples | Contributes 1 training sample |
| Average successful trajectories within each prompt, then average across prompts | First average its 8 successes | First average its 1 success |
The second option is the key structure of MaxRL’s estimator: a prompt with more successes does not automatically receive a larger total coefficient simply because it contributes more retained samples. This does not guarantee equal gradient norms for the two prompts; a prompt with \(K=0\) still provides no successful learning signal in this estimator.
Thus, “filter good videos and train again” is not automatically a video version of MaxRL. A closer implementation requires online sampling, success normalization within each prompt, and an explicit terminal success criterion. If a diffusion video model uses a denoising loss instead of exact generation log-likelihood, that introduces another surrogate objective. LiFT’s existing results do not establish this equivalence to MaxRL.
Back to AR video: bringing the actual feedback loop into training
The three pairings above do not yet directly solve the most common history-distribution problem in AR video. During training, history comes from real videos; during generation, it comes from the model itself. The same parameters can behave very differently under those two conditions.
Self Forcing intervenes most directly: training itself performs autoregressive self-rollouts, making each later chunk depend on previously generated content, and then trains with video-level distribution-matching objectives such as DMD, SiD, or GAN losses. The training objective thus sees the video distribution produced by the actual generation process, rather than only the outputs of local predictions conditioned on real histories.
The related Diffusion Forcing trains with independent noise levels for each token or frame, helping the model adapt to histories of varying reliability. This provides robustness, but adding noise to real histories does not automatically sample the erroneous histories that the model will actually produce.
Self Forcing also does not guarantee that arbitrarily long generation will avoid degradation. Its practical implementation controls cost through gradient truncation, randomly ending denoising early, and detaching gradients through the KV cache. Quality may still decline beyond the training context length. It directly narrows exposure bias without eliminating every numerical bias and every long-horizon error.
Stability must also serve the right objective
Reward Forcing provides a more concrete example: retaining initial frames as a long-term attention sink helps maintain a historical reference, but can also make the model repeatedly copy the initial content, weakening motion or even causing visual flashbacks. It uses EMA-Sink to update this part of the history representation and Re-DMD to give self-rollouts with better motion quality greater weight in distribution matching.
This shows why “more stable” and “better video” must also be distinguished. A nearly motionless video may find it easy to maintain a consistent appearance. Reward Forcing’s reward-weighting mechanism is a broad neighbor of MaxRL, but it does not optimize the same \(\log P(\mathrm{success}\mid c)\) objective or use the same success normalization within each prompt.
AR-CoPO instead focuses on the relationship between exploration and the actual sampler. For few-step AR video, switching to an SDE with intermediate noise injection to make an RL loss easier to compute may move training exploration away from the actual ODE generation process. AR-CoPO perturbs initial noise at a selected pivot chunk, branches into candidate videos, and then makes local updates through a contrastive surrogate policy. It shares BPO’s motivation to reconsider how a policy loss should be constructed, but does not include its Bellman elimination.
Diagnosing a new system through this lens
The most useful outcome of bringing these methods together is not a unified algorithm name, but a map of where to intervene.
| Observed problem | What to locate first | Related methods | What this does not establish |
|---|---|---|---|
| Sampling and training probabilities differ on the same prefix; updates keep drifting | The conditional distribution at a fixed history and its mean score | Score Centering; BPO’s full KL structure | Full recovery of the on-policy gradient |
| Quantizing historical KV shifts attention toward the past | How nonlinear computation turns noise into bias | Quantized Keys Steal Attention | Removal of all history errors or long-video drift |
| The first chunks look normal, but quality degrades when relying on generated history | Visitation distributions of real and generated histories | Self Forcing; Diffusion Forcing | Indefinite stability beyond the training length |
| Overall reward improves little, and the local policy loss is difficult to explain | The derivation from the overall objective to the surrogate loss | BPO; Flow-DPO; AR-CoPO | Mathematical equivalence among the three |
| Easy scenarios succeed repeatedly, while hard scenarios lack usable candidates | Learning weights on prompts and successful trajectories | MaxRL; LiFT’s filtering variant; Reward Forcing | Correction of sampler mismatch or temporal errors |
For a new AR video system, I would begin with three independent checks. First, fix the history and parameters, change only sampling precision, the KV representation, or the compute engine, and look for a systematic shift in conditional outputs. Second, keep the sampler fixed, compare performance under real histories and self-rollout histories, and report degradation curves over generation length. Third, inspect success rates and successful sample counts by prompt to determine whether training repeatedly reinforces only the scenarios that are already easy.
Validation should be separated accordingly: bias correction should be evaluated on the expectation it claims to correct; history-distribution training on the long-term quality of actual rollouts; and success reweighting on coverage of hard prompts and gains under different sampling budgets. A single average reward or overall video score can rarely establish all these mechanisms individually.
A reasonable research plan for combining these ideas would be to first train on actual AR self-rollouts, bringing the history distribution closer to deployment; add targeted corrections for computational biases that have been verified; and then use an objective with a clear trajectory-level meaning to decide how much learning signal each generation receives. This is a design proposal informed by the literature, not a combined system already validated by these papers together.
The shared problem across domains is how a model interacts with the states, data, and learning signals it produces itself. Identifying the common feedback structure helps us transfer ideas; retaining each method’s intervention point and mathematical limits lets us determine what that transfer has actually fixed.
References
This article analyzes and compares mechanisms from public papers; it reports no new model experiments. The gradient connection between BPO and Score Centering is an algebraic observation derived from the original equations. The other cross-domain pairings are analogies between mechanisms or derivation patterns, as specified in the text.
- Score Centering Stabilizes Off-policy Reinforcement Learning. Martin Marek, Max Ryabinin. 2026. arXiv:2609.20807. Main sources: §4, drift decomposition and score centering; Appendix A, the top-\(k\) implementation.
- Bellman Policy Optimization. Zhuoqing Song et al. 2026. arXiv:2609.15987. Main sources: §3.1–3.2, the full trajectory objective, PMD equivalence, and practical approximations.
- Maximum Likelihood Reinforcement Learning. Fahim Tajwar et al. 2026. arXiv:2602.02710, v3. Main sources: §3–4, the truncated objective and success-conditioned estimator.
- Quantized Keys Steal Attention: Bias Correction for KV-Cache Compression in Video Diffusion. Tuna Tuncer et al. 2026. arXiv:2605.26266. Main sources: §4.1–4.2, Jensen bias and additive correction.
- Improving Video Generation with Human Feedback. 2025. arXiv:2501.13918, v2. Main source: §4, Flow-DPO, Flow-RWR, and the design of time weights.
- LiFT: Leveraging Human Feedback for Text-to-Video Model Alignment. 2024 / 2025. arXiv:2412.04814, v3. Main sources: §3.4 and §4.4, reward-weighted learning and the rejection sampling variant.
- Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion. 2025. arXiv:2506.08009, v2. Main sources: §3, self-rollouts and video distribution matching; §5, limitations.
- Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion. 2024. arXiv:2407.01392. Per-token noise training and stability of long rollouts.
- Reward Forcing: Efficient Streaming Video Generation with Rewarded Distribution Matching Distillation. 2025. arXiv:2512.04678, v2. Main source: §3, EMA-Sink and Re-DMD.
- AR-CoPO: Align Autoregressive Video Generation with Contrastive Policy Optimization. 2026. arXiv:2603.17461, v2. Main source: §3, pivot-chunk exploration and the contrastive surrogate policy.