Memory-Conditioned Scene Diffusion-Forcing

MCSDF

the per-token noise level IS the knownness map

observe 18 frames → generate the 63 forward frames (81-frame video)

VGGT 3D memory · geometry noise map · controlled ablation vs AsyncPatch · download .pptx

Goal — extend a real walkthrough beyond what was filmed

Task

Given the START of a real walkthrough (frames + poses), continue the camera forward and synthesize what it would see — coherent NEW content, consistent with the observed scene.

Why hard

The far frames are genuinely unseen: the model must generate geometry, not interpolate — without drift or hallucination.

Success

SOTA on the DL3DV ctx18 benchmark (20 held-out scenes, 9 published methods on the same protocol).

blue = 18 observed cameras · red = 63 targets · ★ far point

Interface — input / output

INPUT

18 RGB frames 480×832 (~3s of walkthrough)
K (3×3 intrinsics)
81 camera poses — context + the forward path to follow
no depth sensor, no scene prior

OUTPUT

81-frame video on the input trajectory
frames 0–17 = observed (passthrough)
frames 18–80 = 63 GENERATED
scored seen vs unseen separately (k-stratified)

Protocol identical to the ctx18 benchmark → direct leaderboard comparison.

Core idea — noise level = knownness

Every latent token gets its own diffusion timestep from how well the 3D memory knows that region: seen → low noise (trust memory) · unseen → pure noise (generate)

knownness k (top) → noise map τ=clip(1−0.95k, 0.45, 1) (bottom). AsyncPatch's spatial-timestep idea → 3D video tokens, driven by geometry.

Pipeline — build a 3D memory, then paint from it

Stage 1 — VGGT consistent geometry

GT / OLD per-frame mono-depth (shattered) / NEW VGGT (coherent)

18 frames → VGGT one multi-view pass → consistent point cloud + poses; Umeyama-aligned to the real cameras (residual ~1.4%).

Old inconsistent depth shattered the memory → the model read fragments as trees → forest hallucination. Fixed.

Stage 2 — render the memory → conditioning signals

z_mem what memory thinks each frame looks like · k where it was seen (cover × photo-consistency × count) · τ=1−λk the noise map

k varies with real coverage (std 0.26; the old leaky cache was flat 0.02)

Validated: corr(k, memory-error) = −0.37…−0.59 over 45 clips — error falls monotonically with k.

Architecture — frozen 14B + 2 trainable parts (+1 optional)

Wan2.1-I2V-14B DiT — 40 blocks, FROZEN (generative prior; geometry lives in the memory)

① LoRA rank 64/128q,k,v,o,ffn — the task adapter

② Trained time-modulation pathtime_embedding + time_projection + per-block AdaLN (186M) — lets the DiT accept a PER-TOKEN timestep field (the AsyncPatch requirement)

③ Memory slots (cross-attn)REMOVED in the current controlled ablation — plain warp skeleton first; re-added later as its own variable

Training: anchor-mixture flow matching x_t=(1−t)c+t·ε, v=(x_t−x0)/t; seen tokens sometimes anchored at z_mem → learn to CORRECT the warp. Sampler: per-token Δt + final x0-projection (+13–22% sharpness).

Evidence so far

far frame 40 — pre-fix hallucinated a forest; all current variants generate the corridor (attribution: memory quality)

k is informative: corr(k, memory-error) −0.37…−0.59; oracle error lights up exactly where k is dark.
Gray-leak diagnosed: output grayness ↔ warp grayness corr +0.92 — empty-warp gray leaks via partially-covered tokens; the noise map is the countermeasure (uniform baseline suffers most).
Sampler correctness: per-token Δt + x0-projection, +13–22% sharpness (A/B validated).

Running now — 3-run decomposition + full-scale baseline (all → 10k)

A-cont vs B — training field: random vs geometry (tests AsyncPatch's claim)

A-mask vs A-cont — inference: binary vs graded (same weights)

baseline r128 vs B — noise map vs plain warp at a capacity disadvantage

Milestones: 3k matched snapshot → videos + k-stratified metrics → 10k → scale winner → ctx18 leaderboard.

live results page →