1ReLER, CCAI, Zhejiang University
Corresponding Author
4D generation seeks to synthesize dynamic 3D scenes from conditions such as text and images. Existing systems either reconstruct decoded RGB videos with a separate 4D model—introducing distribution mismatch and cascading errors—or specialize a video generator for geometry prediction, limiting transfer across backbones and conditioning schemes. We instead treat the terminal denoised latent of video DiTs sharing a VAE as a unified interface to explicit 4D prediction. Latent-to-4D bypasses RGB decoding by aligning this latent with a pretrained 4D decoder and refining it through alternating frame-wise and global spatiotemporal attention. Trained on roughly 1K continuous 81-frame clips from six existing reconstruction datasets, one checkpoint transfers unchanged across three compatible conditional DiTs. On Text4D-200 and I4D-200, it improves DINO-F1 over matched Wan+4RC cascades by 2.88–3.45 and 5.81 points, respectively; multi-view human evaluation also favors its geometry, completeness, and temporal stability.
L4AR maps VAE-space video latents onto the token grid of a pretrained 4D decoder. During training it consumes observed-video latents; at inference the same pathway consumes terminal denoised latents from any compatible DiT.
Without task-specific 4D retraining, the same checkpoint inherits conditioning and control signals from compatible video DiTs. These examples test interface transfer across generation, animation, trajectory control, and action-conditioned world modeling; they do not imply physical or task success.
Text-conditioned terminal latents are lifted into 81-frame cameras and dynamic world-space point maps.
A cinematic photorealistic Stegosaurus crosses a monumental Victorian railway palace. Static wide three-quarter camera, continuous motion, no cuts.
A first-frame image anchors appearance and layout while its I2V latent is lifted into a dynamic 4D scene.
A person holding chopsticks over a bowl of food.
A reference image and driving audio condition facial expression, body motion, and camera dynamics before latent-to-4D lifting.
A man in a suit leans forward on a sofa and speaks with a serious expression of concern.
A reference image and demonstration video specify the target subject and transferable transformation.
Target: the craftsman and workshop transform into cel animation. Reference: the meal undergoes the same transformation.
A character reference and driving performance define identity and motion for the shared 4D pathway.
The person performs the demonstrated motion.
A reference image establishes appearance while a skeleton sequence constrains pose over time.
A brass-and-porcelain robot chef performs a full-body routine while strictly following the pose condition.
Dense point trajectories steer scene motion from an image-conditioned initial state.
Two little girls hold a tea party on a blue sofa; one pours tea for the other in a bright room.
An initial scene and action instruction produce a manipulation latent that the unchanged L4AR pathway maps to a dynamic 4D rollout.
Pick up the blue clothing and place it into the laundry basket.
More compatible video DiT applications remain open for 4D exploration.
Controlled comparisons hold the generated Wan latent fixed: the RGB baseline decodes it before 4RC, while ours lifts it directly. Off-axis renderings expose visible surface support and completeness; they are projection-based evidence, not metric 4D geometry.
Hover or focus the viewer to play; drag to rotate or scrub through all 81 generated frames.
If you find this work useful, please cite:
@misc{liu2026pixelsvideopriors4d,
title = {Beyond Pixels: From Video Priors to 4D Worlds},
author = {Zihao Liu and Xiaolong Shen and Zhenglin Zhou and Ruijie Quan and Yi Yang},
year = {2026},
eprint = {2608.10744},
archivePrefix = {arXiv},
primaryClass = {cs.CV},
url = {https://arxiv.org/abs/2608.10744}
}