FlowVLA Enabling World Dynamics Understanding
through Pluggable 3D Flow Learning

The pipeline of FlowVLA. Given RGB observations o_t−L+1:t, a language instruction l and proprioceptive states s_t−L+1:t, the VLA backbone f_θ produces action latents z_t:t+k−1 that the action policy p_θ decodes into actions a_t:t+k−1. FlowVLA asks the same latents to also explain how the 3D scene will move, through three components. (1) Reconstruction-guided 3D flow construction. A feed-forward dynamic reconstruction model (VDPM) recovers reference-anchored 3D point trajectories P₀(t′, u) for every pixel u of the first frame, from RGB alone. A rigid transform T_base←0 ∈ SE(3) expresses the points in the robot base frame, and the 3D flow ΔP₀(t′, u) = P₀(t′+1, u) − P₀(t′, u) is computed between adjacent frames. The top-K points by average flow magnitude (K = 2048) form the motion-salient keypoint set 𝒰_t, which filters out static background and reconstruction noise without any semantic masks or contact labels. (2) Point-level flow. Each action latent z_t+j is broadcast to the K keypoints, fused with their 3D coordinates, and passed through a flow head that predicts every keypoint's future displacement ΔP̂₀^(j)(t, u); the Smooth-L1 loss ℒ_point grounds the latent in dense scene-level 3D motion. (3) TCP-level flow. A weighting head predicts softmax weights r_t^(j) ∈ ℝ^K over the keypoints, and the weighted sum of the predicted point flows Σ_u r_t^(j)(u) ΔP̂₀^(j)(t, u) must match the ground-truth tool-center-point displacement, giving ℒ_tcp — scene motion distilled into control-relevant motion. The final objective is ℒ_total = ℒ_action + λ_point ℒ_point + λ_tcp ℒ_tcp. All flow heads and 3D geometric inputs are used only during training and discarded at inference, so the deployed policy is architecturally identical to the original VLA and runs through its original 2D pathway with zero inference-time overhead. The flow heads add only 0.34% trainable parameters and about 1.26× training time.

We attach FlowVLA to two VLAs with substantially different action-generation policies — Seer, a two-layer MLP-based policy that reconstructs future observations and predicts actions, and π_0.5, a PaliGemma-based VLM with a flow-matching action policy (LoRA fine-tuned) — and evaluate on the four LIBERO suites (Spatial, Object, Goal, Long) and on LIBERO-Plus, which tests zero-shot robustness under seven perturbation types across 10,030 episodes. For Seer we follow the original LIBERO-Long protocol (top-3 checkpoints, 20 rollouts per task). 3D flow objectives are attached only during training; model architectures and inference pathways are unchanged.

We further evaluate zero-shot robustness under test-time variations that were never seen during training. For Basketball, we change the angle and height of the third-person camera (viewpoint variation); for Ring Insertion, we remove the 20 mm ring support so that the initial ring height changes (height variation).

With only 40 demonstrations per task, the baseline frequently stops execution prematurely, collides, or places objects at incorrect locations under variations of the initial object configuration; these failures become more severe in the 3D-aware tasks and under viewpoint and height variations. FlowVLA shows no premature stopping, grasps more accurately, and completes precise manipulation in most 3D-aware scenarios, except for extreme cases where the target object lies near the camera boundary. Left: Seer; right: Seer + FlowVLA (ours). Clips are keyframe previews from the paper's failure-case figures.