Multi-scale hybrid vision transformer and Sinkhorn tokenizer for sewer defect classification

2022/10/20 by Joakim Bruslund Haurum, Meysam Madadi, Sérgio Escalera +2 · 2 citations
Computer Science · Engineering · #Advanced Neural Network Applications #Digital Media Forensic Detection #Infrastructure Maintenance and Monitoring

paper · pdf · doi:10.1016/j.autcon.2022.104614

openalex publication_date 2022/10/20 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/02

Abstract

A crucial part of image classification consists of capturing non-local spatial semantics of image content. This paper describes the multi-scale hybrid vision transformer (MSHViT), an extension of the classical convolutional neural network (CNN) backbone, for multi-label sewer defect classification. To better model spatial semantics in the images, features are aggregated at different scales non-locally through the use of a lightweight vision transformer, and a smaller set of tokens was produced through a novel Sinkhorn clustering-based tokenizer using distinct cluster centers. The proposed MSHViT and Sinkhorn tokenizer were evaluated on the Sewer-ML multi-label sewer defect classification dataset, showing consistent performance improvements of up to 2.53 percentage points.

Citations

• A defect classification methodology for sewer image sets with convolutional neural networks
• Underground sewer pipe condition assessment based on convolutional neural networks
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
• Automated sewer pipe defect tracking in CCTV videos based on defect detection and metric learning
• Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
• CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification
• DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification
• Sewer-ML: A Multi-Label Sewer Defect Classification Dataset and\n Benchmark
• CoAtNet: Marrying Convolution and Attention for All Data Sizes
• Early Convolutions Help Transformers See Better
• Automatic detection of sewer defects based on improved you only look once algorithm
• Point Transformer
• Exploiting Generative AI to Scale up Intelligent Tutoring Systems

Cited by

• Token Compression Meets Compact Vision Transformers: A Survey and Comparative Evaluation for Edge AI
• Real-time defect detection in underground sewage pipelines using an improved YOLOv5 model