Recurrent Video Masked Autoencoders

Abstract

We present Recurrent Video Masked-Autoencoders (RVM): a novel approach to video representation learning that leverages recurrent computation to model the tempo- ral structure of video data. RVM couples an asymmetric masking objective with a transformer-based recurrent neu- ral network to aggregate information over time, training solely on a simple pixel reconstruction loss. This design yields a highly efficient "generalist" encoder: RVM achieves competitive performance with state-of-the-art video models (e.g. VideoMAE, V-JEPA) on video-level tasks like action classification, and point and object tracking, while matching or exceeding the performance of image models (e.g. DI- NOv2) on tasks that require strong geometric and dense spatial features. Notably, RVM achieves strong performance in the small-model regime without requiring knowledge dis- tillation, exhibiting up to 30× greater parameter efficiency than competing video masked autoencoders. Finally, we demonstrate that RVM’s recurrent nature allows for stable feature propagation over long temporal horizons with lin- ear computational cost, overcoming some of the limitations of standard spatio-temporal attention-based video models. Ablation studies further highlight the factors driving the model’s success, with qualitative results showing that RVM learns rich representations of scene semantics, structure, and motion.

DAVIS Video Segmentation

We present qualitative results on 7 randomly selected videos from the DAVIS-2017 dataset for the video object segmentation task (first-frame ground-truth provided). The task is to propagate the ground-truth object segmentation from the first frame to all subsequent frames.

JHMDB Pose Tracking

We present qualitative results on 5 randomly selected videos from the JHMDB dataset for the human pose tracking task (first-frame ground-truth provided). The task is to propagate the ground-truth human keypoints from the first frame to all subsequent frames.

KMeans Visualization

We present qualitative results on 5 randomly selected videos from the DAVIS-2017 dataset using KMeans clustering to illustrate how each model decomposes visual structure in a video. KMeans is applied directly to the raw feature maps without any additional processing, using K = 5 clusters.

Noise Video Comparison

We present qualitative results on a random noise video using PCA and KMeans clustering to evaluate whether each model’s representations can capture motion independent of semantic content.

PCA Visualization

We present qualitative results on 5 randomly selected videos from the DAVIS-2017 dataset using principal component analysis (PCA) to illustrate what each model primarily captures in a video. We extract the first three principal components and visualize them as RGB images.

VIP Part Propagation

We present qualitative results on 5 randomly selected videos from the VIP dataset for the video part segmentation task (first-frame ground-truth provided). The task is to propagate the ground-truth human part segmentation from the first frame to all subsequent frames.

Libero Benchmark Results (Frozen Backbone)

We evaluate frozen visual representations on the LIBERO robotics manipulation benchmark. We train a single multi-task language-conditioned policy simultaneously across all 10 long-horizon tasks from the Libero-10 suite.

⚙️ Policy Architecture & Multimodal Conditioning

Frozen Visual Encoders: Video and image backbones (ViT-L) are kept strictly frozen. At each timestep, observations from two camera views (3rd-person agent view and wrist eye-in-hand view) over a temporal window (e.g., 2–4 history frames) are encoded into visual tokens.
Language & Proprioception: Task descriptions are encoded via a frozen SigLIP (B/16) text encoder. Low-dimensional robot proprioception states (joint positions and gripper state) are embedded via a 2-layer MLP.
Cross-Attention Conditioning: A 6-layer Transformer Conditioning Processor with 64 learned query tokens cross-attends across spatial visual features from both camera streams, temporal frame embeddings, language tokens, and proprioceptive features.

🎯 Diffusion Policy & Training Setup

Action Predictor: A 6-layer DiT (Diffusion Transformer) decoder predicts continuous action chunks with a prediction horizon of 10 steps and an execution horizon of 8 steps (receding horizon control).
Diffusion Schedule & Sampling: Discrete-time DDPM diffusion process with 50 diffusion timesteps trained with L1 noise prediction loss, evaluated using a deterministic DDIM sampler (50 steps) with static thresholding.
Training: Trained for 100K gradient steps with AdamW (batch size 128, initial lr 1e-4 with warmup and cosine decay to 1e-7, weight decay 1e-6, and EMA decay 0.999).

📊 Evaluation Protocol & Key Takeaways

Rigorous Evaluation: Evaluated in the MuJoCo / Robosuite simulation environment. Results are averaged over 5 independent training seeds, with each evaluation running 25–50 rollout episodes per task.
Spatio-Temporal Features Excel at Manipulation: RVM achieves the highest overall success rate and the most stable convergence across all seeds.
Video vs. Image Backbones: Recurrent video foundation models (RVM, MooG_v2) consistently outperform static image models (e.g., DINOv2) and non-recurrent baselines (CroCo, V-JEPA), highlighting the critical role of temporally coherent spatio-temporal representations for robotic control and physical interaction.

Note: The robotics evaluations featured above are a joint collaboration with Aravindh Mahendran, Stannis Zhou, Meet Dave and Rajkumar Vasudeva Raju.