EgoPush

Egocentric Multi-Object Rearrangement for Mobile Robots via Constrained Teacher Observability

Boyuan An, Zhexiong Wang*, Yipeng Wang*, Jiaqi Li, Sihang Li†, Jing Zhang†, Chen Feng†
New York University
* Equal Contribution, † Corresponding Author
arXiv PDF

TL;DR

EgoPush is a learning framework that enables a mobile robot to perform long-horizon, multi-object non-prehensile rearrangement using only egocentric vision, without relying on global maps, precise localization, or external tracking.


How Do Mobile Robots Rearrange Objects using only Egocentric Vision?

Task Formulation

Task formulation: TurtleBot3 Burger rearranges colored boxes into a cross using only egocentric vision.

EgoPush Deployment in Real-World


Try it your self!

Click start to load Unity WebGL. Clicking outside the Unity area will stop it. If performance is not ideal, open in new tab.

Open in new tab Troubleshooting


Abstract

Humans rearrange objects in cluttered environments using egocentric perception, actively moving to keep task-relevant spatial cues in view. Mobile robots have not matched this: rearrangement is usually built on a global pose estimate or a map, which is exactly what a robot carrying one camera lacks, while pushing keeps changing the scene it would have to be built from. We present EgoPush, which pushes objects into anchor-relative formations from onboard RGB-D alone, with no global localization, external tracking, or map at deployment, and transfers zero-shot to a TurtleBot in controlled and visually cluttered scenes. What makes this learnable turns out to be a property of the teacher rather than of the student: three privileged teachers trained with identical rewards, architecture, and hyperparameters all exceed 98% success, yet their distilled egocentric students reach 0%, 54.8%, and 87.3%, the only variable being the teacher’s observation function. EgoPush therefore trains the teacher under egocentric observability constraints, restricting it to visibility-limited cues and revealing target references only when the anchor is centrally visible, so that its supervision is recoverable by a depth-based student distilled online. Making the teacher trainable in the first place needs two further pieces: a role-grouped object-centric interface shared by teacher and student, and stage-wise temporally decayed rewards for long-horizon credit assignment.


Failure Cases of Baselines

Why Constrain Teacher Observability?

All three teachers exceed 98% success, but their students range from 0% to 87.30%.

Teacher and distilled student success rates. Global observation without FOV: 99.22% and 0%. FOV without center gating: 98.34% and 54.79%. EgoPush with FOV and center gating: 99.31% and 87.30%.
Teacher and student rollouts under the three observation settings.

EgoPush Method

Overview Image

How It Works

1.1 Object-Centric Representation:
(1) Task Roles: EgoPush partitions scene objects into three task roles: active object (currently pushed), anchor object (defines the target relation), and obstacles.
(2) Role-wise Encoding: A shared-weight estimator encodes each role into a latent embedding, and these embeddings are concatenated as an object-centric latent state.

1.2 Relative Spatial Reasoning:
(3) Relation-First Representation: Weight sharing places all roles in a common feature space, enabling the policy to reason over relative spatial relations instead of isolated object states.

Object-Centric Representation

Results in Simulation

Depth Processing and Sim2Real

Comparison of Depth Processing Methods

Results in Real World

Trial 01 · 5×
Trial 02 · 5×
Trial 04 · 5×
Trial 05 · 5×
Trial 06 · 5×
Trial 07 · 5×
Trial 08 · 5×
Trial 09 · 5×
Trial 10 · 5×
Trial 11 · 5×
Trial 12 · 5×
Trial 13 · 5×
Trial 14 · 5×
Trial 15 · 5×
Trial 16 · 5×

BibTeX


@article{An2026EgoPush,
  title = {EgoPush: Egocentric Multi-Object Rearrangement for Mobile Robots via Constrained Teacher Observability},
  author = {An, Boyuan and Wang, Zhexiong and Wang, Yipeng and Li, Jiaqi and Li, Sihang and Zhang, Jing and Feng, Chen},
  journal={arXiv preprint arXiv:2602.18071},
  year = {2026}
}