AutoSpeed: Annotation-Free Stage-Adaptive Motion Speed Learning for Robot Manipulation

We categorize the literature on motion speed modulation in robot manipulation by when the motion speed ratio is inferred: (1) pre-training inference, where it is inferred before training and used to annotate the dataset; (2) test-time inference, where it is inferred during model deployment; and (3) training-time emergence from unlabeled data, where it is implicitly inferred during end-to-end training.

The following methods fall into the first category: DemoSpeedUp [demospeedup] derives safe-to-accelerate regions by estimating action entropy with a proxy policy, and retrains a faster policy on the annotated dataset. Similarly, ESPADA [espada_2512_entropy] uses VLM/LLM-based stage annotations to identify the safe-to-accelerate regions. Moreover, SpeedAug [speedaug] constructs tempo-augmented trajectories to learn a tempo-enriched imitation prior, then applies RL-based fine-tuning to push toward faster policies. And the following methods fall into the second category: SAIL [sail] integrates complexity-aware speed modulation with high-gain tracking and scheduling to safely increase execution speed, while SRIL [SRIL_2410_skipaction] reduces computing latency by learning when to skip inference. VFIL [VFIL_2410] treats speed as an explicit command and conditions the policy on sampling frequency, enabling variable-speed execution by changing the control frequency at deployment without retraining.

However, the third paradigm remains underexplored despite its practical importance. Annotation-based approaches face two key limitations. First, introducing external tools or training additional models to produce annotations is costly and often requires task-specific heuristics or rules. Second, annotation quality and cross-trajectory consistency directly affect the final policy performance, yet these annotations are not optimized end-to-end with the policy. Moreover, prior work largely focuses on when to accelerate; in AutoSpeed, we unify acceleration and deceleration within a single speed-modulation perspective.

Action chunking, which models the joint distribution of future actions conditioned on past states, is a standard practice in robotic manipulation [act, dp]. While action chunking strengthens temporal consistency and mitigates error accumulation, long chunks reduce access to the most recent observations during execution and thus limit reactivity [bid]. A growing body of work shows that the prediction horizon in action chunking can substantially impact policy performance, suggesting the existence of an optimal horizon [autohorizon, bid, rtc]. Embodied manipulation is inherently stage-structured: predictability and feedback requirements vary substantially across stages, so a single global optimal horizon is at best an average-case compromise. A more appropriate view is local optimality, where the effective horizon should be adapted to the current context during policy prediction or execution.

From the training-time perspective, MoH [mixhorizon] formulates action chunking as a mixture over multiple horizons, processes them in parallel with a shared action transformer, and fuses the outputs via a gating module. From the inference-time perspective, AutoHorizon [autohorizon] adjusts the execution horizon online based on the model’s attention weights. GBC [selfguidechunkadaptive] improves diffusion-based behavior cloning via self-guidance adaptive chunking mechanism that selectively refreshes action chunks when higher reactivity is needed.

In contrast to these horizon-adaption designs, AutoSpeed does not explicitly enumerate or select action chunks with different horizons. Instead, the policy is directly optimized toward the optimal action target, avoiding the need to model suboptimal horizon modes or incur the inference-time cost of grouped prediction. Moreover, the speed ratio emerges implicitly during policy learning, making the mechanism model-agnostic and easy to apply across policy architectures.

The shared motivation between motion speed modulation and action chunking reconfiguration is to move beyond a single globally tuned trade-off toward stage-aware, locally optimal adaptation across task stages. Dynamic adjustment of the prediction horizon covered by action chunks is also one of our motivations, and we find it is strongly positively correlated with motion-speed modulation across task stages. Accordingly, we keep the output chunk size fixed to preserve architectural compatibility, and use motion-speed modulation to induce a coupled change in the effective inference horizon, enabling stage-adaptive temporal reasoning without altering the model output dimensionality.

Imitation learning for robotic manipulation aims to train a policy by minimizing the prediction error of demonstrated trajectories conditioned on observations. An expert dataset is given as $\mathcal{D}=\{(l^{(i)},\tau^{(i)})\}_{i=1}^{N},$, where each trajectory $\tau$ is denoted by $\tau=\{(o_{t},a_{t},s_{t})\}_{t=1}^{T}$ and the task instruction $l$ specifies the task. At each time step $t$, $o_{t}$ denotes the image observation, $a_{t}$ denotes the motion control signal, and $s_{t}$ denotes the robot state. We denote the chunk size by $H$, and the observation horizon by $K$. For brevity, we omit conditioning inputs other than $\mathbf{o}_{t-K+1:t}$. A standard visuomotor policy $\pi_{\theta}$ predicts a length-$H$ action chunk conditioned on the recent observation context:

To enable stage-adaptive motion speed learning, the key challenge is to identify which stages in demonstration trajectories can be safely accelerated. We define the motion speed ratio as $r_{t}$ at time step $t$. Larger $r_{t}$ corresponds to easier stages, allowing faster execution and a longer temporal prediction horizon; smaller $r_{t}$ corresponds to precision-critical stages, demanding slower execution, a shorter horizon and higher-frequency feedback.

In our framework we use a fixed number of chunks to keep the model’s output tensor shape consistent. Therefore, the temporal prediction horizon $h_{t}$ is positively correlated with the motion speed ratio $r_{t}$. then:

Accordingly, a visuomotor policy trained with AutoSpeed is formulated as:

where $\mathbf{a^{\prime}}_{t:t+h_{t}-1}$ has a motion speed $r_{t}$ matched to the current task stage. The transformation from $\mathbf{a}_{t:t+H-1}$ to $\mathbf{a^{\prime}}_{t:t+{h_{t}}-1}$ is described in Sec. 3.3. Besides, Sec. 3.2 details the AutoSpeed optimization procedure. Sec. 3.4 presents training recipes for generative policy classes and introduces a compatible inference-time strategy that aggregates overlapping predictions to improve smoothness. In the following sections, we denote an action chunk by $A$.

Recent studies have analyzed and empirically validated quantifiable stage signatures in robotic manipulation: fine-grained behaviors are often reflected in the high-frequency components of action trajectories[FreqPolicy], while the policy’s predicted action entropy can serve as a proxy for identifying difficult, failure-prone stages[25nips_entropy_FIPER, demospeedup, rss25MIentropy]. We hypothesize that the stage-dependent signal $r_{t}$ can be implicitly inferred during end-to-end training, and in this work we validate that it is feasible and model-agnostic. We formulate training as cost-aware multi-target optimization.

We apply the discrete cosine transform (DCT) [dct-theory] to map time-domain action signals into the frequency domain, perform temporal scaling there to realize stretching/compression along the time axis, and then transform the actions back to the time domain.

In details, we first compute DCT-II coefficients of the original action chunk $A\in\mathbb{R}^{H_{0}\times D}$ along the temporal axis, where $H_{0}$ denotes the maximum number of action samples required:

We then retime by evaluating the inverse-DCT basis at the speed ratio $r$ and reconstruct the retimed chunk via a basis–coefficient product:

where $t_{i}=i\cdot r$ for $i=0,\dots,H-1$, $k=0,\dots,H_{0}-1$, $\alpha_{0}=1/\sqrt{2}$, and $\alpha_{k}=1$ for $k\geq 1$. During training, the candidates $\{\tilde{A}^{(m)}\}_{m=1}^{M}$ are generated by applying the operator with the speed ratio set $\{r_{m}\}_{m=1}^{M}$.

This enables both acceleration and deceleration with non-integer scaling factors, while the frequency-domain processing effectively preserves high-frequency details that are crucial for fine-grained manipulation[FreqPolicy].

We evaluate AutoSpeed across a range of standard manipulation benchmarks and real-world tasks.

Table 1 summarizes the quantitative results evaluated on the ALOHA benchmark.

Compared to the vanilla ACT baseline, the policy trained with AutoSpeed (denoted as AutoSpeed) achieves a substantial reduction in task execution time without compromising overall performance. Specifically, on the Transfer Cube task, AutoSpeed reduces the average episode length from 272 to 160 steps, yielding a maximum speedup of 1.7x. While this accelerated execution slightly lowers the success rate (from 72% to 64%) under standard control due to the heightened sensitivity of high-speed interactions, coupling AutoSpeed with a high-gain controller effectively mitigates this issue. This combination not only preserves the execution efficiency but also surpasses the original baseline with a 6% improvement. Furthermore, on the contact-rich Insertion task, AutoSpeed simultaneously improves the success rate (from 22% to 24%) and reduces execution length (from 353 to 296 steps with high-gain), demonstrating its robust execution efficiency in single-task learning scenarios.

The quantitative results on the LIBERO-10 and Meta-World benchmark are presented in Table 2. AutoSpeed consistently outperforms the strongest baselines across both multi-task benchmarks. On Meta-World, AutoSpeed yields substantial improvements in success rates regardless of the underlying action head: BAKU-DiT exhibits a 19.0% increases, while BAKU-Flow improves by 7.8% to achieve the highest overall success rate (65%). Crucially, these gains are accompanied by marked reductions in average episode lengths. On the LIBERO-10 suite, AutoSpeed-trained policy compresses the execution time with a maximum acceleration of approximately 40% to its vanilla counterpart. Compared to baselines that naively execute actions at a fixed, dataset-dictated speed, AutoSpeed’s stage-adaptive optimization empowers policies to complete tasks substantially faster, while simultaneously improving success rates by dynamically adjusting to the complexity of the current task phase.

Crucially, our experiments demonstrate that reducing task execution time and improving success rates are not mutually exclusive. By predicting trajectories at stage-adaptive motion speeds, AutoSpeed leverages long-horizon action chunks for faster execution during easy stages, where future actions can be predicted reliably. Conversely, during fine-grained stages that demand tighter perception-action coupling, the framework naturally adopts slower execution and shorter effective horizons to preserve control accuracy. Because this stage-aware modulation is implicitly inferred during end-to-end training via our composite cost, AutoSpeed generalizes effectively across multi-task scenarios, delivering superior execution efficiency while preserving task success compared to baselines.

Beyond qualitative observations, we further analyze the consistency between the predicted speed ratios and action entropy from DemoSpeedUp [demospeedup], which reflects divergence across multiple sampled action predictions, and find that unlike action entropy, which relies on grouped predictions from a proxy policy and struggles to distinguish uncertainty from multi-modality, AutoSpeed more accurately identifies safely acceleratable stages, such as robot start-up and return-to-home fast motions.

As shown in Table 3, AutoSpeed consistently outperforms the baselines across all evaluated real-world tasks with various length and difficulty levels. Most notably, when integrated with our proposed Nonlinear Temporal Aggregation (NTA), the AutoSpeed-trained policy significantly reduces execution time, delivering an average speedup of approximately 1.78$\times$ across the diverse task suite.

Crucially, this substantial acceleration does not compromise manipulation precision but yields consistent absolute improvements in task success rates. As illustrated in Fig. 4, in the highly challenging Place the Toy task, AutoSpeed increases the success rate by a notable 10%. These physical deployments further validate AutoSpeed’s capacity to autonomously extract stage-dependent motion speed patterns from unannotated demonstrations. By leveraging the proposed multi-target optimization, the policy translates these learned patterns into superior task efficiency and robust real-world performance.

Notably, AutoSpeed trained on non-oversampled data still outperforms the baselines, experiencing only a marginal performance drop compared to the oversampled version. Since action oversampling is practically feasible, it simply serves as an optional enhancement to further improve fine-grained action fidelity enabled by DCT-based retiming.