What Actually Matters? A Paired Evaluation of Inference Settings for π0.5 on BEHAVIOR-1K

What it does. The policy predicts a chunk of 30 actions per query. The execution horizon is how many of them the robot executes before the policy is queried again. The actions not executed overlap with the next chunk and seed its inpainting. The baseline executes 26 of the 30, played back in 20 control steps.

Why we test it. The 2025 BEHAVIOR winner predicts 30 actions, executes 26 and keeps the last 4 for inpainting. In limited ablations, the authors report that small variations in the number of executed actions did not produce significant changes, whereas executing only 10 actions from a 30-action chunk led to score degradation. Comet replans after executing the full predicted action segment and reports that this receding-horizon strategy outperforms temporal ensembling and receding-temporal execution, which smooth or average open-loop predictions. On the single “turning on radio” task, Comet reports a non-monotonic relationship between action horizon and success: the success rate peaks at 32 steps (0.30) and declines at 50 steps (0.25). In contrast, π0 predicts 50-action chunks and executes a fixed prefix before requesting a new observation: 16 actions for the 20 Hz UR5e and Franka robots, and 25 actions for its other robots operating at 50 Hz. The ablation in Diffusion Policy found that an action horizon of eight steps performed best for most tested tasks, reflecting a trade-off between action consistency and responsiveness to new observations.

Conclusion. Executing only 13 actions per query clearly reduces Q-score and success rate while increasing episode length. Executing all 30 actions without overlap, by contrast, produces no measurable change. These results suggest that action consistency matters more than responsiveness to new observations in our setting. We hypothesise that each query samples a fresh noise seed, potentially leading consecutive chunks to pursue different strategies. Halving the execution horizon increases the frequency of chunk boundaries, while the four inpainted actions may only smooth transitions between chunks. Moreover, the policy was fine-tuned on its own rollouts, all collected with a 26-of-30 execution prefix. At 13 actions, it may therefore encounter states from which it was not trained to initiate a chunk. Responsiveness is less critical in our setting because the simulator waits for the policy to produce its next action and the scenes are largely static.

What it does. The playback speed-up sets how many control steps the executed actions are spread over, so the robot moves faster than the demonstrations were recorded. The baseline plays 26 actions back in 20 control steps (1.3×). We tested 26 in 26 steps (1×, demonstration pace) and 26 in 13 steps (2×).

Why we test it. The 2025 BEHAVIOR winner plays each chunk back faster than it was recorded and reports the gain only qualitatively. It also reports that doubling the speed degraded its score. SAIL studies faster-than-demonstration execution for imitation policies and uses adaptive speed modulation to slow down around precision-critical phases, identified in part through gripper opening and closing events, while accelerating simpler motions.

Conclusion. Slowing playback from 1.3× to demonstration speed (1×) reduces performance and increases episode length, whereas speeding it up to 2× produces no measurable change. One likely contributor at 1× is the challenge's 1.5× time limit: slower playback needs 1.3× as many steps for the same motions, so episodes that succeed close to the limit no longer finish in time. This accounts for only part of the drop, though, because only about one in ten successful episodes comes that close to the limit. At 2×, any benefit of faster execution may be offset by less precise manipulation, particularly around gripper events, as reported by SAIL.

What it does. The action expert is a flow-matching model that turns a noise sample into an action chunk over a fixed number of denoising steps, each one forward pass of the action expert. More steps integrate the flow more finely.

Why we test it. π0 and π0.5 generate action chunks using 10 flow-matching integration steps at inference time to fit a real-robot latency budget. GR00T N1 uses four flow-matching inference steps, which the authors report worked well across embodiments. The 2025 BEHAVIOR winner code we inherited uses 20. Because the simulator waits for the policy, denoising steps cost compute but no latency here, so the only question is whether more steps give better actions.

Conclusion. Neither 1 nor 10 denoising steps measurably changed the results. We hypothesise that one step already produces sufficiently accurate actions for this policy. Frequent replanning may compensate for less accurate predictions by correcting errors from new observations, while additional steps may primarily refine actions that are never executed before replanning. Thus, the inherited 20-step sampler may incur substantially higher computational cost without measurable gains in task success.

What it does. Overlap inpainting joins consecutive chunks: the last 4 actions of the previous chunk, not yet executed, are held fixed and the new chunk is generated around them, so the motion does not jump at the chunk boundary. Off generates each chunk from scratch.

Why we test it. Overlap inpainting is the 2025 BEHAVIOR winner’s own scheme for joining chunks and was never ablated. Real-Time Chunking addresses a related problem by generating chunks asynchronously, freezing actions guaranteed to be executed and guiding the remaining overlapping actions to improve continuity despite inference delay. It reports improved task throughput and robustness to inference delay. Because our simulator waits for the policy, our experiment isolates chunk joining from latency compensation.

Conclusion. Turning overlap inpainting off did not change the result measurably. We hypothesise that with 26 of 30 actions executed, the four overlapping actions are too few to matter and the policy’s chunks are already consistent at the boundary. The latency compensation Real-Time Chunking was designed for is not exercised in a simulator that waits for the policy.

What it does. During training, every action chunk carries a label that says whether it came from a good or a bad episode. Some of the time the label is hidden, so the model also learns to act without one. At inference we always show it the good label and ask for the actions a good episode would contain. The guidance strength β controls how strongly that label is applied. The model forms two action proposals at each denoising step, one with the good label and one without. The difference between them is what the label changes. β = 1 uses the labelled proposal as it is, which is the plain conditional policy. β = 2 doubles the difference and β = 4 quadruples it, pushing the actions further toward what the model associates with good episodes than it would go on its own.

Why we test it. π*0.6 / RECAP uses β=1 by default and relies primarily on training-time advantage thresholds to obtain a well-conditioned policy. The authors also explore moderate guidance strengths (e.g., β ∈ [1.5,2.5]) where useful, cautioning that high values can push actions toward the boundary of the learned distribution and produce overly aggressive motions. LeHome uses substantially higher classifier-free guidance scales (β=7-9), selected through online Thompson-sampling bandit optimization rather than controlled ablations. The search began in the range (0-2), but repeatedly favored higher values, prompting the author to expand the search range.

Conclusion. β = 2 hurts slightly and β = 4 shows no measurable change. Thus, stronger guidance provides no observable benefit in our experiments. We hypothesise that because positive labels cover demonstrations, successful episodes, and productive prefixes, the conditional and unconditional policies may differ too little for guidance to improve action predictions substantially. Alternatively, stronger guidance may distort the predicted action distribution without improving task performance.

What it does. The robot has three cameras, one on the head and one on each wrist. The policy looks at each camera image at a size of 224 × 224 pixels. The demonstrations the policy was trained on were recorded at a larger size, 720 × 720 for the head camera and 480 × 480 for the wrist cameras, and then shrunk to 224 × 224. The baseline does the same at inference: it renders at the large size and shrinks the image, so the policy sees the same kind of image it saw in training. The tested setting renders at 224 × 224 directly, which is cheaper. The two images are not identical, though. When a large image is shrunk, each small pixel is the average of several large ones, so edges look soft and thin objects blend slightly into their surroundings. A direct small render samples one point per pixel, so edges look crisp and jagged and very thin objects can fall between pixels. The policy was trained only on shrunk images, so a direct render shows it a slightly different style of picture of the same scene.

Why we test it. Comet reports improved performance when rendering observations at the cameras’ original resolutions rather than directly at 224 × 224. We adopted original-resolution rendering followed by downsampling after observing a non-significant trend in a single census. This test checks whether that choice yields a measurable benefit under our paired protocol. The policy always receives 224 × 224 images Only the rendering pipeline differs: either the simulator renders directly at 224 × 224, or it renders at the demonstrations’ original resolutions (720 × 720 for the head camera and 480 × 480 for the wrist cameras) before the wrapper downsamples the images.

Conclusion. Direct 224 × 224 did not change the result significantly, although Q-score and success rate both point slightly downward. If the drop is real, a likely cause is that rendering at the demonstrations' original resolutions and then downsampling better matches the visual inputs seen during training. Direct low-resolution rendering would then introduce a small visual distribution shift, because the policy was trained on downsampled high-resolution renders rather than images rendered directly at 224 × 224.

What it does. The gripper snap is a correction rule at inference. The simulator releases a grasped object only when the gripper command reaches exactly 1.0, while the policy’s open commands often land just below it and silently keep holding. The snap projects open-side commands above 0.9 onto 1.0, the value the demonstrations contain. It is on in the baseline and known to help, so turning it off is a positive control.

Why we test it. The gripper snap came out of watching rollout videos: the robot would reach the goal position, for example hovering over the trash can with the trash in hand, and then not let go. It would retreat, approach again and hover again, still holding the object. The recordings showed that the simulator only releases when the gripper command reaches exactly 1.0, while the policy’s open commands typically stopped just short of it.

Conclusion. Turning the snap off lowered success rate, in line with its known effect, while the drop in Q was not significant. The snap helps on specific tasks, mainly those that need repeated grasp-and-release, such as putting objects into a bin or sink. On the other tasks it makes little difference. Because those grasp-and-release tasks are a minority of the test panel, the overall effect is small, whereas during collection it was measured on train instances with many attempts on exactly those tasks.

What it does. At every query the policy samples 4 action chunks in one batched pass, each from its own noise sample, and executes one of them. The baseline samples a single chunk. The chunk is chosen by consensus: the four chunks are compared pairwise with dynamic time warping, chunks closer than the median pairwise distance are grouped, and the medoid of the largest group, the chunk closest to the others in it, is executed. The executed chunk is always one of the samples, never an average of them, so it stays a trajectory the policy actually produced.

Why we test it. Stochastic action generation makes inference-time sampling a natural way to improve action selection without retraining the policy, an idea well studied as best-of-N sampling for language models (Gui et al., 2024; Huang et al., 2025). Recent work on VLA action-chunk selection demonstrates that sampling multiple candidates and selecting among them can improve manipulation performance: VGAS reports an increase in LIBERO-Goal success from 52.0% with single-sample inference to 60.2% with four candidates and value-guided selection (Xu et al., 2026). Related work on Expected-Max Q-Learning likewise shows that selecting among a small number of sampled actions can improve performance over single-sample policy execution (Ghasemipour et al., 2021). Motivated by these results, we test whether a similar benefit can be obtained for our flow-matching action expert without an additional learned scorer.

Conclusion. Best-of-4 consensus selection yielded no measurable improvement over single-sample inference, while leaving episode length unchanged and increasing sampling cost approximately fourfold. Several factors may explain this result: the sampled action chunks may differ numerically without producing meaningfully different trajectories. Another idea is that failures may be dominated by systematic policy errors and perception or contact uncertainties that additional sampling cannot resolve. Thus, in the current setting, consensus-based selection does not demonstrate sufficient benefit to justify its additional inference cost.

What it does. The policy is fine-tuned on its own rollouts, and only the successful ones, or the productive part of a partial one, become training data. Most rollouts fail, and a failed episode that got far before failing has already paid for a lot of simulation. Starting the instance fresh throws that work away. A restart instead resumes the failed episode from a snapshot of the simulator state saved while it was still going, so the policy only has to redo the part after the snapshot. The goal is more successful episodes per simulation step. The restart point sets which snapshot is used. At last progress is the state at the moment the episode last made progress, that is, when it last satisfied a new task predicate. Before last progress is a state a few hundred steps earlier, so the policy has to approach that milestone again rather than starting right on it. Before first drop is a state from before the first step at which the policy’s predicted progress score fell, which marks its first mistake. The last two points can lie after that mistake, since in most failures a milestone still follows it, so only this snapshot starts before the error.

Why we test it. LeHome uses a learned success head to detect potential failure states, saves them for later replay, and retries from those states. The paper separately reports improved robustness from replaying successful states with stronger visual augmentations. Go-Explore archives promising states, returns to selected states before exploring further, and can restore simulator states directly. Reverse Curriculum Generation progressively expands the training start-state distribution from states near the goal to increasingly distant states.

Per restart point. Final Q includes credit already held at the snapshot. “Q earned after restart” excludes it.

Conclusion. No restart point succeeds more often than a fresh start. Every restart point does end with a higher final Q, though, because a restarted episode keeps the partial credit the failed episode had already earned and the fresh start has to earn everything again. Restarts also need far fewer simulation steps per episode, since the part before the snapshot is not simulated again. Among the restart points, restarting before the first drop works best: it succeeds more often than restarting before or at the last progress and earns the most progress after the restart. We hypothesise that the snapshot of a failed episode already carries the cause of the failure, such as a misplaced object or an awkward grasp, so resuming it recovers little, while an earlier restart leaves more of the task to be earned.

What it does. This test compares two estimates of the same quantity: how much more often a failed episode succeeds when it is restarted from a snapshot than when the task instance is started fresh. The first estimate comes from the collection logs. During collection, a scheduler we call the planner decides what happens after an episode fails: restart it from the snapshot taken just before the policy’s progress score first dropped, or start the task instance fresh. It bases that decision on what it knows about the episode. We label this estimate planner’s choice. It is observational: the planner, not chance, decided which episodes were restarted, so the two groups can differ in more than the restart. The second estimate comes from our randomised experiment.

Why we test it. The planner’s choice estimate may be biased because the planner uses information about an episode to decide whether to restart it. For example, restarted episodes may have been more likely to succeed even without a restart. Comparing this observational estimate with the randomised estimate helps us assess whether the apparent benefit reflects the restart strategy itself or the planner’s selection of episodes.

Conclusion. In the collection logs, the planner’s restarts before the score drop succeeded more often than fresh episodes, by +0.075 ± 0.028. When the same failed episodes were continued both ways in the randomised experiment, the difference was +0.003 ± 0.025, so restarting itself gives no detectable advantage in success rate. The two estimates differ by 0.072 with a standard error of 0.038, which is suggestive rather than decisive. The simplest explanation is selection: the planner chose which episodes to restart, and we hypothesise that it favoured tasks and instances the policy can solve, where any second attempt succeeds more often. Regression to the mean after a noisy score drop would add to the apparent gain. Which episodes were picked for a restart therefore mattered more than where the restart happened.

What it does. A snapshot restore puts the simulator back to the saved state. Our policy carries state of its own. Following the 2025 BEHAVIOR winner, it tracks a task stage, a coarse index of progress through the task. The 2025 winner split each demonstration into 5 to 15 stages by time, while this year the dataset provides the stage labels. A classifier on the VLM output predicts the current stage at every query. The predictions are smoothed by a vote over the last three: the belief advances when two agree on the next stage and falls back only when all three agree on the previous one. The voted stage is encoded into stage tokens and fed back as an input, so a faithful restore hands the policy this stage belief and its vote buffer along with the simulator state. We test two departures from the faithful restore. The first drops the stage belief, so the policy restarts at stage 0 and has to re-estimate its stage from what it sees. The second keeps the stage belief but offsets the robot’s base pose from the saved one by 0.15 m and 15°. The scene and the stage belief then no longer match the robot’s position exactly, so the policy has to absorb the offset in its first chunks.

Why we test it. Prior work restores saved simulator states for replay or exploration (LeHome, Go-Explore), while Reverse Curriculum Generation progressively expand training starts from near the goal to more distant states. Comet perturbs initial robot poses during rejection-sampling fine-tuning and retains successful rollouts for training, improving overall validation performance. Our policy feeds its voted task stage back as an input, so a snapshot holds more than the simulator state. We tested a restore without the stage belief and one with the robot’s base pose perturbed, each paired with the faithful restore of the same snapshot.

Conclusion. Dropping the stage belief lowers success rate, final Q and earned progress, so the stage belief is part of the state a restart must restore. Perturbing the pose is not resolved by the panel. We hypothesise that without its stage belief the policy re-enters an earlier stage and repeats work the snapshot had already completed, or skips checks it believes done, whereas a small pose perturbation stays within the variation the demonstrations cover.