CNN-ViT Hybrid for Pneumonia Detection: Theory and Empiric on Limited Data without Pretraining

2025/09/10 by Prashant Singh Basnet, Basnet, Prashant Singh, Roshan Chitrakar +1
Computer Science · Medicine · #COVID-19 diagnosis using AI #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #FOS: Electrical engineering #Image and Video Processing (eess.IV) #Imbalanced Data Classification Techniques #Machine Learning in Healthcare #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2509.08586

openalex publication_date 2025/09/10 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

This research explored the hybridization of CNN and ViT within a training dataset of limited size, and introduced a distinct class imbalance. The training was made from scratch with a mere focus on theoretically and experimentally exploring the architectural strengths of the proposed hybrid model. Experiments were conducted across varied data fractions with balanced and imbalanced training datasets. Comparatively, the hybrid model, complementing the strengths of CNN and ViT, achieved the highest recall of 0.9443 (50% data fraction in balanced) and consistency in F1 score around 0.85, suggesting reliability in diagnosis. Additionally, the model was successful in outperforming CNN and ViT in imbalanced datasets. Despite its complex architecture, it required comparable training time to the transformers in all data fractions.

Citations

• Exploring the Synergies of Hybrid CNNs and ViTs Architectures for Computer Vision: A survey
• The harm of class imbalance corrections for risk prediction models: illustration and simulation using logistic regression
• The harm of class imbalance corrections for risk prediction models: illustration and simulation using logistic regression
• MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer
• Do Vision Transformers See Like Convolutional Neural Networks?
• Going deeper with Image Transformers
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Performance Analysis and Comparison of Machine and Deep Learning Algorithms for IoT Data Classification
• CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and\n Expert Comparison
• Global Weighted Average Pooling Bridges Pixel-level Localization and Image-level Classification
• Deep Convolutional Neural Networks for Image Classification: A Comprehensive Review
• Attention Is All You Need
• Deep Residual Learning for Image Recognition
• An Introduction to Convolutional Neural Networks
• Visualizing and Understanding Convolutional Networks
• ImageNet classification with deep convolutional neural networks
• Power laws, Pareto distributions and Zipf's law
• Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
• Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

Related

• Diagnostic Uncertainty in Pneumonia Detection using CNN MobileNetV2 and CNN from Scratch
• Comparative Analysis of Vision Transformers and Traditional Deep Learning Approaches for Automated Pneumonia Detection in Chest X-Rays
• Deep Learning-Based Pneumonia Detection from Chest X-ray Images: A CNN Approach with Performance Analysis and Clinical Implications
• FMT:A Multimodal Pneumonia Detection Model Based on Stacking MOE Framework
• A novel method to enhance pneumonia detection via a model-level ensembling of CNN and vision transformer