Learning Attribute-aware Representations for Few-shot Scene Text Segmentation

2025/04/15 by Chenming Li, Yifan Tang, Li, Chenming +14
Computer Science · #Advanced Neural Network Applications #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Handwritten Text Recognition Techniques #Multimodal Machine Learning Applications #cs.CV

paper · pdf · doi:10.48550/arxiv.2504.11164

openalex publication_date 2025/04/15 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28 · arxiv created 2026/08/04 · arxiv updated 2026/08/05

Abstract

Supervised scene text segmentation has achieved notable progress in recent years. However, its development is largely constrained by the scarcity of high-quality datasets and the high cost of pixel-level annotations. To address this limitation, we explore few-shot learning for text segmentation and propose TSAL, an attribute-aware few-shot framework that leverages a pre-trained CLIP model to learn transferable text attributes for segmentation. Our framework comprises two complementary branches: I) a Visual-Guided Branch that extracts semantic and textural features for foreground text and background regions, respectively, and II) an Adaptive Prompt-Guided Branch that employs learnable prompt templates to capture diverse text attributes with minimal data dependence. To effectively align textual attributes with visual representations, we further introduce an Adaptive Feature Alignment~(AFA) module, which aligns learnable attribute tokens with visual features and prompt prototypes, enabling the model to capture both general and distinctive textual characteristics. As a result, TSAL can accurately segment text regions using only a few annotated samples. Extensive experiments demonstrate that our method achieves state-of-the-art performance across several public text segmentation benchmarks under few-shot settings and exhibits strong generalization to text-related tasks.

Citations

• FG-CLIP: Fine-Grained Visual and Textual Alignment
• MSDNet: Multi-Scale Decoder for Few-Shot Semantic Segmentation via Transformer-Guided Prototyping
• WAS: Dataset and Methods for Artistic Text Segmentation
• EAFormer: Scene Text Segmentation with Edge-Aware Transformers
• Rethinking Prior Information Generation with CLIP for Few-Shot Segmentation
• FiLo: Zero-Shot Anomaly Detection by Fine-Grained Description and High-Quality Localization
• PromptAD: Learning Prompts with only Normal Samples for Few-Shot Anomaly Detection
• Hi-SAM: Marrying Segment Anything Model for Hierarchical Text Segmentation
• BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editing
• SegGPT: Segmenting Everything In Context
• WinCLIP: Zero-/Few-Shot Anomaly Classification and Segmentation
• Hierarchical Dense Correlation Distillation for Few-Shot Segmentation
• CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentation
• Weakly-Supervised Text Instance Segmentation
• ZegCLIP: Towards Adapting CLIP for Zero-shot Semantic Segmentation
• CLIP-ReID: Exploiting Vision-Language Model for Image Re-Identification without Concrete Text Labels
• LAION-5B: An open large-scale dataset for training next generation image-text models
• Learning Non-target Knowledge for Few-shot Semantic Segmentation
• Learning What Not to Segment: A New Perspective on Few-Shot Segmentation
• Conditional Prompt Learning for Vision-Language Models
• Language-driven Semantic Segmentation
• Image Segmentation Using Text and Image Prompts
• VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts
• LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs
• Learning to Prompt for Vision-Language Models
• Vision Transformer for Fast and Efficient Scene Text Recognition
• Learning Transferable Visual Models From Natural Language Supervision
• Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
• ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision
• Rethinking Text Segmentation: A Novel Dataset and A Text-Specific Refinement Approach
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Prior Guided Feature Enrichment Network for Few-Shot Segmentation
• A Simple Framework for Contrastive Learning of Visual Representations
• How Can We Know What Language Models Know?
• Weak Supervision for Generating Pixel-Level Annotations in Scene Text Segmentation
• VisualBERT: A Simple and Performant Baseline for Vision and Language
• Neural Poetry: Learning to Generate Poems using Syllables
• Total-Text: A Comprehensive Dataset for Scene Text Detection and Recognition

Related

• Beyond Visual Cues: Leveraging General Semantics as Support for Few-Shot Segmentation
• CoAPT: Context Attribute words for Prompt Tuning
• Textual and Visual Guided Task Adaptation for Source-Free Cross-Domain Few-Shot Segmentation
• Text and Image Are Mutually Beneficial: Enhancing Training-Free Few-Shot Classification with CLIP
• Task-Adaptive Feature Transformer for Few-Shot Segmentation