HMVLM: Human Motion-Vision-Lanuage Model via MoE LoRA

2025/11/03 by Lei Hu, Hu, Lei, Yongjing Ye +3
Computer Science · Engineering · #68T45 #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Graphics (cs.GR) #Human Motion and Animation #Human Pose and Action Recognition #I.2.10 #I.3.7 #Multimodal Machine Learning Applications

paper · pdf · doi:10.48550/arxiv.2511.01463

openalex publication_date 2025/11/03 · openalex created_date 2025/11/06 · openalex updated_date 2026/07/28

Abstract

The expansion of instruction-tuning data has enabled foundation language models to exhibit improved instruction adherence and superior performance across diverse downstream tasks. Semantically-rich 3D human motion is being progressively integrated with these foundation models to enhance multimodal understanding and cross-modal generation capabilities. However, the modality gap between human motion and text raises unresolved concerns about catastrophic forgetting during this integration. In addition, developing autoregressive-compatible pose representations that preserve generalizability across heterogeneous downstream tasks remains a critical technical barrier. To address these issues, we propose the Human Motion-Vision-Language Model (HMVLM), a unified framework based on the Mixture of Expert Low-Rank Adaption(MoE LoRA) strategy. The framework leverages the gating network to dynamically allocate LoRA expert weights based on the input prompt, enabling synchronized fine-tuning of multiple tasks. To mitigate catastrophic forgetting during instruction-tuning, we introduce a novel zero expert that preserves the pre-trained parameters for general linguistic tasks. For pose representation, we implement body-part-specific tokenization by partitioning the human body into different joint groups, enhancing the spatial resolution of the representation. Experiments show that our method effectively alleviates knowledge forgetting during instruction-tuning and achieves remarkable performance across diverse human motion downstream tasks.

Citations

• Temporal Continual Learning with Prior Compensation for Human Motion Prediction
• DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning
• Multi-Transmotion: Pre-trained Model for Human Motion Prediction
• MotionGPT-2: A General-Purpose Motion-Language Model for Motion Generation and Understanding
• MotionFix: Text-Driven 3D Human Motion Editing
• MotionLLM: Understanding Human Behaviors from Human Motions and Videos
• Motion-Agent: A Conversational Framework for Human Motion Generation with LLMs
• M³GPT: An Advanced Multimodal, Multitask Framework for Motion Comprehension and Generation
• Flexible Motion In-betweening with Diffusion Models
• TokenHMR: Advancing Human Mesh Recovery with a Tokenized Pose Representation
• MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Experts
• MotionChain: Conversational Motion Controllers via Multimodal Prompts
• Diffusion-based Human Motion Style Transfer with Semantic Guidance
• MoELoRA: Contrastive Learning Guided Mixture of Experts on Parameter-Efficient Fine-Tuning for Large Language Models
• AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling
• X-LoRA: Mixture of Low-Rank Adapter Experts, a Flexible Framework for Large Language Models with Applications in Protein Mechanics and Molecular Design
• DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models
• MMM: Generative Masked Motion Model
• ChatPose: Chatting about 3D Human Pose
• MoMask: Generative Masked Modeling of 3D Human Motions
• Video-LLaVA: Learning United Visual Representation by Alignment Before Projection
• LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset
• Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuning
• NExT-GPT: Any-to-Any Multimodal LLM
• Probabilistic Triangulation for Uncalibrated Multi-View 3D Human Pose Estimation
• AttT2M: Text-Driven Human Motion Generation with Multi-Perspective Attention Mechanism
• MotionGPT: Human Motion as a Foreign Language
• MotionGPT: Finetuned LLMs Are General-Purpose Motion Generators
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models
• Humans in 4D: Reconstructing and Tracking Humans with Transformers
• Guided Motion Diffusion for Controllable Human Motion Synthesis
• AMD: Autoregressive Motion Diffusion
• Visual Instruction Tuning
• ReMoDiffuse: Retrieval-Augmented Motion Diffusion Model
• GPT-4 Technical Report
• Human Motion Diffusion as a Generative Prior
• LLaMA: Open and Efficient Foundation Language Models
• T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representations
• Executing your Commands via Motion Diffusion in Latent Space
• Human Motion Diffusion Model
• Neural3Points: Learning to Generate Physically Realistic Full-body Motion for Virtual Reality Users
• MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model
• TM2T: Stochastic and Tokenized Modeling for the Reciprocal Generation of 3D Human Motions and Texts
• TEMOS: Generating diverse human motions from textual descriptions
• MotionCLIP: Exposing Human Motion Generation to CLIP Space
• HuMoR: 3D Human Motion Model for Robust Pose Estimation
• Learning Transferable Visual Models From Natural Language Supervision
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
• End-to-End Human Pose and Mesh Reconstruction with Transformers
• HybrIK: A Hybrid Analytical-Neural Inverse Kinematics Solution for 3D Human Pose and Shape Estimation
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Character Controllers Using Motion VAEs
• Local motion phases for learning multi-contact character movements
• Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loop
• Generating Multiple Hypotheses for 3D Human Pose Estimation with Mixture Density Network
• Generalizing Monocular 3D Human Pose Estimation in the Wild
• Mode-adaptive neural networks for quadruped motion control
• Combining Recurrent Neural Networks and Adversarial Training for Human Motion Synthesis and Control
• Phase-functioned neural networks for character control
• Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
• Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Image
• Learning Factored Representations in a Deep Mixture of Experts
• Adaptive Mixtures of Local Experts
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Related

• Being-M0.5: A Real-Time Controllable Vision-Language-Motion Model
• Human Motion Instruction Tuning
• MotionGPT3: Human Motion as a Second Modality
• LangPose: Language-Aligned Motion for Robust 3D Human Pose Estimation
• TWIST & SCOUT: Grounding Multimodal LLM-Experts by Forget-Free Tuning