Memory-Conditioned Scene Diffusion-Forcing

MCSDF

per-token noise level = knownness · AsyncPatch for posed video

input 18 frames → generate 63 forward frames (81-frame video)

Current pipeline · VGGT geometry · forward ctx18 · LoRA-128

Real forward scene extrapolation

Given

the first 18 frames of a real walkthrough + their camera poses.

Produce

the 63 forward frames the camera moves into — regions it has not seen.

Why hard

the far frames are genuinely unseen: generate coherent new geometry, don't copy. Same protocol as the ctx18 benchmark (train = eval).

camera path — blue = 18 context (seen), red = 63 targets (generated)

Core idea — per-token noise = knownness

Every latent token carries its own diffusion timestep, set by whether the memory has seen it.

Seen → start at LOW noise from memory content (trust it). Unseen → start at PURE noise (generate freely).

This is AsyncPatch Diffusion for posed video: one joint denoiser reads the whole latent + a spatial timestep field, so seen (clean) tokens condition unseen (noisy) ones via attention.

The knownness map k is derived from 3D geometry (co-visibility + photometric agreement) — a geometry→noise interface, not arbitrary.

Measured: k predicts where the memory is reliable (corr(k, memory-error) = −0.37 to −0.59 over 45 clips).

Architecture — frozen 14B + 3 trainable parts

Wan2.1-I2V-14B DiT · 40 blocks · FROZEN (14B params)

① LoRA rank 128q,k,v,o,ffn — alpha 128 (scaling 1.0)

② Trained modulation pathtime_embedding + time_projection + per-block modulation (186M) — lets the DiT accept a PER-TOKEN timestep (AsyncPatch)

③ Memory modules (every 5 blocks)MemoryRead: 3D-pos-keyed cross-attn to a 4096-slot point bank + confidence-gated shortcut

Backbone frozen throughout — only ①②③ train (~1.1B params). Geometry comes from the memory, not the backbone.

Pipeline — end to end

① Contextfirst 18 frames + poses

→

② VGGT geometryconsistent point cloud, Umeyama→DL3DV

→

③ Memory signalsz_mem · knownness k · 4096 slots

→

④ DiffusionWan-14B + LoRA + mem, noise = k

→

⑤ Output81 frames: 0-17 obs, 18-80 gen

Training objective anchor-mixture Diffusion-Forcing: x_t = (1−t)·c + t·ε , target v = (x_t − x0)/t

For seen tokens the anchor c = z_mem (teaches the model to CORRECT the memory's error); else c = x0.

Sampler per-token Δt + final x0-projection (+13–22% sharpness, integrator-correct).

Key fix 1 — consistent geometry (VGGT)

GT / OLD per-frame mono-depth (shattered) / NEW VGGT (coherent)

Problem

Per-frame mono-depth is inconsistent → points misalign → z_mem is a shattered mosaic → hallucination (walkway → forest).

Fix

VGGT: one multi-view pass → cross-frame-consistent points, Umeyama-aligned to DL3DV (residual ~1.4%).

Result

z_mem is a coherent corridor → the model extends a walkway.

Key fix 2 — forward ctx18 → knownness becomes meaningful

knownness k — blue = unseen (generate), yellow = seen (trust memory)

Before

Memory spanned the whole trajectory → nothing truly unseen → k flat (std 0.02). Degenerate.

After

Memory only from 18 context frames → far region genuinely unseen → k VARIES (std 0.26).

Validated useful

corr(k, memory-error) = −0.37…−0.59; error drops monotonically as k rises.

Status — what is running now

Geometry	VGGT consistent + Umeyama-aligned — validated
Task / schedule	forward ctx18: observe 18, generate 63 (matches eval)
Knownness map	varies (std 0.26); predicts memory reliability
Sampler	per-token Δt + x0-projection (shipped, +13–22%)
Capacity	LoRA rank 128 (alpha 128), 10k steps, 8-GPU DDP
Data	DL3DV 1410-clip train split — re-caching (forward)

Next: finish forward re-cache → launch rank-128 / 10k training → score on the ctx18 leaderboard.