DeepSoMo: Deformable Object Motion Estimation under Proprioceptive Partial Observability via Predictive Latent Dynamics

1New York University, *Corresponding author
DeepSoMo teaser image.

DeepSoMo analyzes only the proprioceptive visual inputs at various time steps to estimate the motion states of a soft ball moving on unknown terrains, including the 6D pose trajectory, the full-body surface shape in the body frame, and the surface points in contact with the terrain.

Abstract

Estimating the motion of deformable bodies in soft robotics from proprioceptive sensing is challenging due to partial observability, as internal observations provide only indirect access to global motion, contact, and shape evolution. In such settings, instantaneous observations are insufficient, and accurate estimation requires temporally consistent latent representations.

This paper studies deformable motion estimation under proprioceptive partial observability from a latent dynamics perspective. We argue that a key requirement for long-horizon estimation is latent temporal consistency, where latent states remain predictive of future observations. To this end, we present DeepSoMo, a learning-based framework that enforces predictive structure in latent dynamics through a self-supervised latent prediction constraint.

We evaluate the approach in a canonical deformable dynamics setting using high-fidelity finite-element simulation as a controlled source of ground truth. Results demonstrate consistent improvements in trajectory estimation, orientation recovery, shape reconstruction, and contact inference over architecture-matched recurrent baselines.

Pipeline

DeepSoMo teaser image.

Results

Main Results

Method Prediction NTE ↓ ATE ↓ ARE ↓ CPC ↑ CD ↓
ResLSTM NoPred 8.3 ± 0.3 56.8 ± 1.1 19.3 ± 1.0 99.0 3.1
ObsPred Image 7.7 ± 0.5 49.9 ± 7.2 19.0 ± 0.8 99.1 2.9
DeepSoMo Latent 6.6 ± 0.4 38.1 ± 3.7 16.8 ± 0.7 99.2 2.5

NTE in %; ATE in cm; ARE in deg.; CPC in % accuracy; CD in units of 10−1 mm. Mean ± std over 3 seeds.

Object Variation

Experiment Method NTE ↓ ATE ↓ ARE ↓ CPC ↑ CD ↓
Canonical sphere ResLSTM 8.3 ± 0.3 56.8 ± 1.1 19.3 ± 1.0 99.0 3.1
DeepSoMo 6.6 ± 0.4 38.1 ± 3.7 16.8 ± 0.7 99.2 2.5
Shape variation ResLSTM 7.5 ± 0.3 37.1 ± 3.4 11.4 ± 0.1 98.9 4.1
DeepSoMo 6.8 ± 0.8 31.5 ± 2.8 11.1 ± 1.1 99.0 3.5
Shape + material ResLSTM 9.7 ± 0.3 57.1 ± 4.3 21.5 ± 0.5 99.1 3.2
DeepSoMo 7.7 ± 0.6 35.6 ± 6.4 19.6 ± 1.6 99.5 1.6

NTE in %; ATE in cm; ARE in deg.; CPC in % accuracy; CD in units of 10−1 mm. Mean ± std over 3 seeds.

Video