Data Modalities

• RGB (.png), depth, IR, IMU, temperature frames compressed in multiple channels in Matroska format (.mkv)
  • RGB color frames (.png) extracted from the Matroska file
  • RGB videos (.mp4) of the color frames from each recording
• Point clouds (.ply) of each scene, produced using AzureKinfu
• Labeled hand/action frames (.txt)
• Hand pose inference results using Google MediaPipe Hands (.txt)

• 15 household scenes
• 15 point cloud files (one for each scene)
• 150 total recordings(10 recordings in each scene, with different object configurations in each recording)
• 15000 hand-object actions (100 per recording)
• ~600 min of RGBD video (~4 min per video)
• ~1,080,000 RGB frames at 30 fps
• ~900,000 hand action frames (assuming ~2 seconds per hand-object action)