Video2STL: Grounding VLM-Generated Temporal Specifications for Robot Learning


Abstract

Video-based policy learning is particularly promising because it illustrates target behaviors without requiring action annotations or embodiment-matched demonstrations. A central challenge is deciding what information should be transferred from the video to the robot, as existing approaches often make the temporal structure of a task difficult to inspect, ground, and reuse. We present Video2STL, a framework that converts observation-only videos into parametric Signal Temporal Logic (STL) specifications and uses the resulting formal representation for robot learning. A vision-language model first extracts an embodiment-independent semantic event trace and then constructs a bank of symbolic temporal specifications. The model determines the task structure, while numerical predicate thresholds and temporal bounds are grounded from successful robot trajectories. For policy learning, short-horizon specifications provide dense rewards through rolling-window quantitative robustness, while a causal monitor over a retained long-horizon specification provides one-time progress rewards for valid temporal prefixes. Across four manipulation tasks, Video2STL achieves 85.8% average success-once and 67.0% success-at-end, compared with 81.5%/59.5% for native dense PPO and 65.0%/42.3% for Text2Reward. In quadruped locomotion, Qwen-3.8 and GPT-5.6-based Video2STL policies achieve 100% success across velocities from 0.3 to 2.1 m/s while remaining competitive in high-speed energy efficiency.


Video2STL Algorithm Pipeline



Methodology

Our framework separates symbolic task inference from embodiment-specific grounding. The pipeline consists of four main stages:

1. Event Extraction

2. VLM STL Generation

3. Target-Embodiment Grounding & Expert-Consistency Filtering

4. Two-Timescale Temporal Reward


Experimental Results

Quadruped Locomotion

Evaluated on Google’s Barkour vb quadruped in MuJoCo XLA (MJX) across a range of commanded forward velocities. Policies are evaluated using 20 independent rollouts of 500 simulation steps. Survival and velocity-tracking success are reported as percentages (higher is better), and CoT denotes cost of transportation (lower is better).

vx (m/s) Video2STL (GPT-5.6) Video2STL (Qwen 3.8) Text2Reward Heuristic
Surv. ↑Succ. ↑CoT ↓ Surv. ↑Succ. ↑CoT ↓ Surv. ↑Succ. ↑CoT ↓ Surv. ↑Succ. ↑CoT ↓
0.3100%100%2.63100%100%2.63100%100%0.91100%100%1.20
0.5100%100%1.80100%100%1.86100%100%0.80100%100%1.00
1.0100%100%1.26100%100%1.37100%100%0.74100%100%1.00
1.6100%100%1.12100%100%0.92100%100%0.87100%100%1.30
1.9100%100%1.13100%100%0.98100%100%1.00100%100%1.40
2.0100%100%1.13100%100%1.00100%100%1.07100%5%1.40
2.1100%100%1.15100%100%1.02100%100%1.14100%0%1.40


Robot Manipulation

Evaluated in ManiSkill3 using native task success conditions across 128 episodes. We report “Success Once” (the condition is reached at least once) and “Success at End” (the condition is met in the final state).

Task Video2STL Text2Reward Native Dense PPO
Succ. Once ↑Succ. End ↑ Succ. Once ↑Succ. End ↑ Succ. Once ↑Succ. End ↑
PushCube0.950.930.650.231.000.84
StackCube0.980.950.980.960.730.52
LiftPegUpright0.590.050.970.500.980.47
PlaceSphere0.910.750.000.000.550.55



Quadruped Locomotion Videos

Below are qualitative evaluations for different reward formulations across low (0.4 m/s), medium (1.2 m/s), and high (2.0 m/s) velocity commands.

Video2STL (GPT-5.6)

vx = 0.4 m/s vx = 1.2 m/s vx = 2.0 m/s

Video2STL (Qwen 3.8)

vx = 0.4 m/s vx = 1.2 m/s vx = 2.0 m/s

Video2STL (Gemini)

vx = 0.4 m/s vx = 1.2 m/s vx = 2.0 m/s

Text2Reward

vx = 0.4 m/s vx = 1.2 m/s vx = 2.0 m/s

Heuristic Baseline

vx = 0.4 m/s vx = 1.2 m/s vx = 2.0 m/s

Robot Manipulation Videos

PushCube

Video2STL Text2Reward Native Dense PPO

StackCube

Video2STL Text2Reward Native Dense PPO

LiftPegUpright

Video2STL Text2Reward Native Dense PPO

PlaceSphere

Video2STL Text2Reward Native Dense PPO

Citation