2024/06/07 by Jason J. Yoo, Yoo, Jason, He, Yingchen +8 · 1 citation
Computer Science · #Advanced Data Compression Techniques #Advanced Image Processing Techniques #Image and Signal Denoising Methods
paper · pdf · doi:10.48550/arxiv.2406.04814
This work demonstrates that training autoregressive video diffusion models from a single video stream\unicodex2013resembling the experience of embodied agents\unicodex2013is not only possible, but can also be as effective as standard offline training given the same number of gradient steps. Our work further reveals that this main result can be achieved using experience replay methods that only retain a subset of the preceding video stream. To support training and evaluation in this setting, we introduce four new datasets for streaming lifelong generative video modeling: Lifelong Bouncing Balls, Lifelong 3D Maze, Lifelong Drive, and Lifelong PLAICraft, each consisting of one million consecutive frames from environments of increasing complexity.