Abstract

Recent advances in robot foundation models have highlighted the potential of in-context learning (ICL) for adapting robot behavior from demonstrations without further fine-tuning. Previous work has used retrieval-based post-training to equip pretrained vision-language-action (VLA) models with the capability for ICL by teaching them to condition action predictions on retrieved visually-relevant demonstration chunks. However, the effectiveness of retrieval hinges on selecting context relevant to the current execution stage, which visual similarity alone may not capture. We introduce VICTR, a framework enabling in-context learning in VLAs via progress-informed context retrieval built around an In-Context Value Function Estimator (IC-VFE). IC-VFE conditions on a reference demonstration video, a language instruction and the current observation to estimate task-relative progress. By conditioning progress estimation on a reference demonstration, IC-VFE can align the current state with a demonstrated execution even for tasks unseen during training. VICTR uses the estimated progress to select demonstration chunks at corresponding stages of execution and supplies them to a π_0.5 policy post-trained for in-context action generation. On unseen out-of-domain real-world manipulation tasks, VICTR raises the success rate from 6.2% for a fine-tuned π_0.5 baseline to 23.8% and nearly doubles the task progress rate, from 28.9% to 55.5%.

Why does context matter?

The real world rarely repeats itself. Objects change, scenes get cluttered, and familiar skills are recombined into new tasks. A useful robot has to keep adapting, the way a person can pick up a new task after watching it done once or twice. Fine-tuning for every new situation is slow and has to be repeated whenever the task or environment changes. In-context learning takes another route: demonstrations of the new task are given to the policy as context, and its weights stay fixed.

Today's VLAs are not built for this. A fine-tuned π_0.5 does well on tasks from its training domain but often fails out of domain, for example with novel objects or new task compositions. VICTR gives π_0.5 demonstration context during both training and deployment. At deployment, the new task is learned from a handful of demonstrations in context, with no parameter updates.

How VICTR works

VICTR has two parts. An In-Context Value Function Estimator (IC-VFE) estimates task progress against a reference demonstration, and the demo chunk at the same progress is retrieved from a demonstration pool. An in-context VLA (IC-VLA) built on π_0.5 takes that chunk as context and generates the next action chunk. Neither model is updated at deployment.

Building the demonstration pool

For each task, a pool of 5 to 10 demonstrations is collected at deployment time and cut into one-second chunks matching the VLA's action horizon (H = 30 at 30 Hz). One demonstration serves as IC-VFE's reference. Each chunk is keyed by its progress v̂_i and a DINOv2 embedding h_i.

In-context value function estimation

Instead of asking which demo frame looks most like the current one, VICTR asks how far along the task the robot is. IC-VFE reads the reference demonstration as context and predicts the current progress v̂_t ∈ [0, 1] as a distribution over 201 bins. Because IC-VFE infers progress from the demonstration in its context instead of relying on tasks memorized in training, it can estimate progress on unseen tasks without fine-tuning, helping the policy adapt to them.

Each step retrieves the chunk with the closest progress,

The estimates are not perfect, but they track the rollout: progress rises as the robot moves through the task and drops when an attempt fails. The retrieved chunks, in turn, generally show what the robot should do next.

Visual similarity is ambiguous: similar-looking frames can come from different stages, and distractors shift the embedding. Retrievals from our evaluation, by signal:

The attention pattern in IC-VFE

Each reference frame is compressed to 16 summary tokens per camera; the current observation keeps all 256 tokens per view. The 18-layer Gemma 3 270M decoder uses hybrid attention: context blocks attend within themselves, and at layers 6, 12 and 18 block-causal attention lets each block read the ones before it. The query block reads all context, and a learned value token reads everything. Hover a cell to see its attention.

Evaluations

We evaluate on a bimanual robot with 16 real-world tasks: 8 in-domain and 8 out-of-domain, where the out-of-domain tasks use novel objects, new skill compositions or semantically new tasks. Each policy gets 10 rollouts per task. Solid bars show binary task success rate; the lighter extension shows task progress, partial credit for completed milestones.

π_0.5: fine-tuned baseline without context. Vision: retrieves context by DINOv2 visual similarity (the RICL recipe). VICTR: retrieves by IC-VFE progress. VICTR_vision: combines the progress and DINOv2 visual distances, each normalized across the pool.

Per-task results

Analysis

Does context improve online value estimation?

We compare each value model's predictions with human-annotated progress curves. IC-VFE agrees best of all the online estimators (models that can run during a rollout). Removing the reference demonstration roughly halves its agreement.

Does better value estimation give a better policy?

We keep the IC-VLA fixed and swap the value estimator used for retrieval at deployment. With IC-VFE, out-of-domain progress reaches 62.7%, against 29.2% for the best alternative.

How many demonstrations are needed?

Performance improves with pool size. Out of domain, it is still rising at ten demonstrations.

Must training and deployment signals match?

Each policy generally does best with the retrieval signal it was trained on. Matching alone is not enough, though: out of domain, the vision-trained policy with matched vision retrieval still falls well short of VICTR with IC-VFE at both training and test.

Citation