Preliminaries
We build on causal, block-wise video generators such as Self-Forcing, which
distill a bidirectional video diffusion transformer into a few-step causal generator. A video is produced
as a stream of 3-frame blocks; each latent frame is a 30×52 grid of 1,560 tokens, and a block is
denoised conditioned only on a bounded, sliding KV cache covering 6 latent frames — one persistent
sink frame, two prior frames, and the current block. The bound keeps per-block compute constant and
enables arbitrary-length generation, but it also forgets by construction: once a token scrolls past the
window it is gone, and the opening frame is its sole survivor. As more of the early visual context is
discarded, the model retains progressively less information about the original appearance; small
inconsistencies accumulate over time, gradually causing the generated content to drift from its earlier
identity.
What counts as new?
A long video does not need its timeline; it needs its cast.
An entity's appearance is established by a sparse set of novel appearance events — its entry and
the later sightings of newly revealed parts — and nothing about the intervening forty seconds is
required to render it faithfully to keep consistency. This raises a natural question: what counts as
new? The answer has two exhaustive cases: at
t = 0 nothing has been seen, so the opening frame is new in its entirety; afterwards,
content is new only relative to what has already appeared — an object entering the frame, a
region disoccluded by motion, a style introduced by a change of scene.
The first frame: a reinforced sink
The base model already keeps frame 0 as an attention sink, but under the default attention
distribution later queries assign insufficient weight to it: the sink mainly anchors global appearance
characteristics — color tone, background, overall style — and helps prevent quality
degradation, while the identity of subjects introduced in the opening scene can still drift. We
therefore strengthen the sink attention, promoting it from a stability anchor to a persistent
first-impression memory that is more strongly attended to by queries, by adding a fixed positive
bias to the pre-softmax logits of the sink keys:
αj = softmaxj (
q⊤kj / √d +
βj ), βj = ln λ for sink keys, 0
otherwise.
Adding ln λ multiplies each sink key's unnormalized attention weight by λ,
uniformly raising the sink's prior importance while preserving content-based selection through
query–key similarity. Because the bias is applied before the softmax, the distribution stays
normalized — sink keys still compete with all other keys — and it is directly compatible
with fused attention kernels. Its magnitude is a measurement rather than a choice: we set
λ = 5, with its effect analyzed in the λ sweep in Results.
Every frame after: an appearance novelty bank
The reinforced sink covers the opening scene, but a long video continually introduces new content: a
second subject enters the frame, a camera pan reveals a storefront, or an occluded region becomes
visible. After t = 0, “new” therefore means content that has not been
observed before, making novelty detection a property of the video stream rather than of the model's
current query. Optical flow provides a direct cue, and admission is query-agnostic: patches are retained
by visual appearance novelty rather than query–key affinity, without entity grouping or
re-identification. The bank capacity K is a configurable memory budget rather than an
architectural constant: one frame's worth of tokens (K = 1,560) works well for the
one-minute videos in our experiments, and longer videos can naturally allocate additional slots. For
each newly decoded frame, RAFT forward–backward flow runs against its predecessor: a pixel
u is newly appeared when its backtrace finds no reliable antecedent — the cycle residual
ecyc exceeds 2 px, the photometric residual epho exceeds 0.2, or
the backtrace leaves the frame. Such pixels score
ν(u) = ecyc(u) +
α epho(u)(α = 0.5; all other pixels
score 0)
and a patch takes the maximum novelty over its pixels, normalized by its block's typical firing
strength and frozen from then on:
s(p) = maxu ∈ p
ν(u), ŝ(p) = s(p) / max(ρb,
1)(ρb = 90th percentile of positive scores in block
b)
This block-wise normalization prevents brief bursts of large motion from dominating the fixed budget,
so admissions depend on relative novelty rather than absolute motion magnitude. The map of
already-claimed novelty is propagated with the flow, so continuously traceable content fires only once,
when it first appears; if a region returns after its flow correspondence has been broken, it can
register as a new appearance event and receive a new anchor.
Each newly decoded frame is compared with its predecessor using RAFT to obtain a novelty map, from
which newly appearing patches are selected (Figure 2a). For each selected patch, we copy the
corresponding key/value vectors from every attention layer of the KV cache and store them with
their frozen novelty scores. At retrieval, bank keys use temporal rotary phase 0 while retaining their
original spatial phase, avoiding unseen long-range temporal RoPE offsets and treating the bank as a
time-neutral appearance memory. The bank is updated with a simple top-K rule: among all
candidates observed so far, only the K highest-scoring patches are retained. Memory retention is
therefore time-neutral and parameter-free: a patch remains in the bank according to its appearance
novelty rather than how recently it was observed, without a learned controller or an explicit recency
term.
Hyperparameters
| Hyperparameter | Value |
| Sink size | 1 latent frame |
| Attention window | 6 latent frames |
| Block size | 3 latent frames |
| Bank capacity K | 1,560 tokens |
| Sink reinforcement λ | 5 |
| Flow cycle-error threshold | 2 px |
| Photometric-error threshold | 0.2 |
| Photometric weight α | 0.5 |
| Score normalization | per-block 90th pct. |