Parse Graph-Based Visual-Language Interaction for Human Pose Estimation

2025/09/09 by Shibang Liu, Liu, Shibang, Xuemei Xie +3
Computer Science · Engineering · #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Human Pose and Action Recognition #Multimodal Machine Learning Applications #Robot Manipulation and Learning

paper · pdf · doi:10.48550/arxiv.2509.07385

openalex publication_date 2025/09/09 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Parse graphs boost human pose estimation (HPE) by integrating context and hierarchies, yet prior work mostly focuses on single modality modeling, ignoring the potential of multimodal fusion. Notably, language offers rich HPE priors like spatial relations for occluded scenes, but existing visual-language fusion via global feature integration weakens occluded region responses and causes alignment and location failures. To address this issue, we propose Parse Graph-based Visual-Language interaction (PGVL) with a core novel Guided Module (GM). In PGVL, low-level nodes focus on local features, maximizing the maintenance of responses in occluded areas and high-level nodes integrate global features to infer occluded or invisible parts. GM enables high semantic nodes to guide the feature update of low semantic nodes that have undergone cross attention. It ensuring effective fusion of diverse information. PGVL includes top-down decomposition and bottom-up composition. In the first stage, modality specific parse graphs are constructed. Next stage. recursive bidirectional cross-attention is used, purified by GM. We also design network based on PGVL. The PGVL and our network is validated on major pose estimation datasets. We will release the code soon.

Citations

• Probabilistic Prompt Distribution Learning for Animal Pose Estimation
• Learning Structure-Supporting Dependencies via Keypoint Interactive Transformer for General Mammal Pose Estimation
• RTMO: Towards High-Performance One-Stage Real-Time Multi-Person Pose Estimation
• LAMP: Leveraging Language Prompts for Multi-person Pose Estimation
• BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
• Referring Image Matting
• ViTPose: Simple Vision Transformer Baselines for Human Pose Estimation
• CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIP
• HRFormer: High-Resolution Transformer for Dense Prediction
• Low-resolution Human Pose Estimation
• AP-10K: A Benchmark for Animal Pose Estimation in the Wild
• Bottom-Up Human Pose Estimation Via Disentangled Keypoint Regression
• An Adversarial Human Pose Estimation Network Injected with Graph Structure
• Learning Transferable Visual Models From Natural Language Supervision
• Multi-Instance Pose Networks: Rethinking Top-Down Pose Estimation
• Distribution-Aware Coordinate Representation for Human Pose Estimation
• Cross-Domain Adaptation for Animal Pose Estimation
• ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
• MirrorGAN: Learning Text-to-image Generation by Redescription
• Deep High-Resolution Representation Learning for Human Pose Estimation
• CrowdPose: Efficient Crowded Scenes Pose Estimation and A New Benchmark
• Simple Baselines for Human Pose Estimation and Tracking
• YOLOv3: An Incremental Improvement
• Vision-and-Language Navigation: Interpreting visually-grounded\n navigation instructions in real environments
• Learning Feature Pyramids for Human Pose Estimation
• Attention Is All You Need
• Stacked Hourglass Networks for Human Pose Estimation
• VQA: Visual Question Answering
• Microsoft COCO: Common Objects in Context

Related

• Refinement Module based on Parse Graph for Human Pose Estimation
• PoseLLM: Enhancing Language-Guided Human Pose Estimation with MLP Alignment
• Human Pose Estimation with Spatial Contextual Information
• GraphMLP: A Graph MLP-Like Architecture for 3D Human Pose Estimation
• Graph-Based 3D Multi-Person Pose Estimation Using Multi-View Images