Tiny but Mighty: A Software-Hardware Co-Design Approach for Efficient Multimodal Inference on Battery-Powered Small Devices

2025/09/25 by Yilong Li, Shuai Zhang, Li, Yilong +12
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Distributed #FOS: Computer and information sciences #FOS: Electrical engineering #Parallel #Signal Processing (eess.SP) #Speech and dialogue systems #and Cluster Computing (cs.DC) #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2510.05109

openalex publication_date 2025/09/25 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Large Multimodal Models (LMMs) are inherently modular, comprising vision and audio encoders, a projector, and a language backbone. Yet existing systems execute them monolithically, underutilizing the heterogeneous accelerators (NPUs, GPUs, DSPs) on modern SoCs and inflating end-to-end latency. We present Nanomind, a hardware-software co-design inference framework that decomposes each LMM into modular "bricks"--vision, projector, language, and audio--and maps each brick to its best-suited compute units. A Token-Aware Buffer Manager (TABM) enables zero-copy embedding transfer across accelerators on unified-memory SoCs, bypassing CPU bottlenecks. Combined with customized hardware, a battery-aware scheduler, and fused low-bit GEMM kernels, Nanomind runs entirely on a compact, battery-powered prototype that operates fully offline. Nanomind reduces end-to-end energy by 42.3% against mainstream edge frameworks and devkits; in its on-demand low-power mode, the prototype runs LLaVA-OneVision-Qwen2-0.5B with a camera for nearly 18.8 hours on a single 2,000 mAh battery.

Citations

• Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
• SmolVLM: Redefining small and efficient multimodal models
• BitNet a4.8: 4-bit Activations for 1-bit LLMs
• PalmBench: A Comprehensive Benchmark of Compressed Large Language Models on Mobile Platforms
• Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
• The Llama 3 Herd of Models
• Fast On-device LLM Inference with NPUs
• PowerInfer-2: Fast Large Language Model Inference on a Smartphone
• PLeak: Prompt Leaking Attacks against Large Language Model Applications
• Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
• The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits
• Improved Baselines with Visual Instruction Tuning
• Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond
• MMBench: Is Your Multi-modal Model an All-around Player?
• ProPILE: Probing Privacy Leakage in Large Language Models
• MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models
• Sigmoid Loss for Language Image Pre-Training
• The case for 4-bit precision: k-bit Inference Scaling Laws
• Robust Speech Recognition via Large-Scale Weak Supervision
• InfographicVQA
• Learning Transferable Visual Models From Natural Language Supervision
• DocVQA: A Dataset for VQA on Document Images
• Level Playing Field for Million Scale Face Recognition
• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration

Related

• SmolVLM: Redefining small and efficient multimodal models
• Modality Inflation: Energy Characterization and Optimization Opportunities for MLLM Inference
• Characterizing Mobile SoC for Accelerating Heterogeneous LLM Inference
• Fast On-device LLM Inference with NPUs
• Compute Can't Handle the Truth: Why Communication Tax Prioritizes Memory and Interconnects in Modern AI Infrastructure