Latent Representations for Visual Proprioception in Inexpensive Robots

2025/04/20 by Sheikholeslami, Sahara, Bölöni, Ladislau
#Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Robotics (cs.RO)

paper · doi:10.48550/arxiv.2504.14634

Abstract

Robotic manipulation requires explicit or implicit knowledge of the robot's joint positions. Precise proprioception is standard in high-quality industrial robots but is often unavailable in inexpensive robots operating in unstructured environments. In this paper, we ask: to what extent can a fast, single-pass regression architecture perform visual proprioception from a single external camera image, available even in the simplest manipulation settings? We explore several latent representations, including CNNs, VAEs, ViTs, and bags of uncalibrated fiducial markers, using fine-tuning techniques adapted to the limited data available. We evaluate the achievable accuracy through experiments on an inexpensive 6-DoF robot.

Citations

• Precision-Focused Reinforcement Learning Model for Robotic Object Pushing
• OpenVLA: An Open-Source Vision-Language-Action Model
• RoboKeyGen: Robot Pose and Joint Angles Estimation via Diffusion-based 3D Keypoint Generation
• Real-time Holistic Robot Pose Estimation with Unknown States
• Markerless Camera-to-Robot Pose Estimation via Self-supervised Sim-to-Real Transfer
• Semi-Perspective Decoupled Heatmaps for 3D Robot Pose Estimation from Depth Maps
• Single-view robot pose and joint angle estimation via render & compare
• Nothing But Geometric Constraints: A Model-Free Method for Articulated Object Pose Estimation
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Camera-to-Robot Pose Estimation from a Single Image
• Accept Synthetic Objects as Real: End-to-End Training of Attentive Deep Visuomotor Policies for Manipulation in Clutter
• CRAVES: Controlling Robotic Arm with a Vision-based Economic System
• Vision-Based Multi-Task Manipulation for Inexpensive Robots Using End-To-End Learning from Demonstration
• Deep Residual Learning for Image Recognition
• Very Deep Convolutional Networks for Large-Scale Image Recognition

Related

• Learning an internal representation of the end-effector configuration space
• Do You Need Proprioceptive States in Visuomotor Policies?
• Lossless Adaptation of Pretrained Vision Models For Robotic Manipulation
• Proprioceptive Image: An Image Representation of Proprioceptive Data from Quadruped Robots for Contact Estimation Learning
• Learning a generative model for robot control using visual feedback