MolmoMotion: Language-guided 3D motion forecasting

MolmoMotion is a new model that can predict the future 3D movement of an object simply by tracking specific points on its surface. Using a single video frame, the points' initial positions, and a natural language instruction, it forecasts the object's trajectory over the next few seconds. To enable this, researchers developed MolmoMotion-1M, the largest dataset of its kind, by automatically extracting 3D motion data and action descriptions from over a million videos. This powerful forecasting ability has significant applications in robotics planning and creating physically plausible, controllable video generation.