Harnessing Diffusion-Generated Synthetic Images for Fair Image Classification

2025/11/11 by Abhipsa Basu, Basu, Abhipsa, Gupta, Aviral +4
Computer Science · #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Face and Expression Recognition #Face recognition and analysis #Generative Adversarial Networks and Image Synthesis

paper · pdf · doi:10.48550/arxiv.2511.08711

openalex publication_date 2025/11/11 · openalex created_date 2025/11/14 · openalex updated_date 2026/07/28

Abstract

Image classification systems often inherit biases from uneven group representation in training data. For example, in face datasets for hair color classification, blond hair may be disproportionately associated with females, reinforcing stereotypes. A recent approach leverages the Stable Diffusion model to generate balanced training data, but these models often struggle to preserve the original data distribution. In this work, we explore multiple diffusion-finetuning techniques, e.g., LoRA and DreamBooth, to generate images that more accurately represent each training group by learning directly from their samples. Additionally, in order to prevent a single DreamBooth model from being overwhelmed by excessive intra-group variations, we explore a technique of clustering images within each group and train a DreamBooth model per cluster. These models are then used to generate group-balanced data for pretraining, followed by fine-tuning on real data. Experiments on multiple benchmarks demonstrate that the studied finetuning approaches outperform vanilla Stable Diffusion on average and achieve results comparable to SOTA debiasing techniques like Group-DRO, while surpassing them as the dataset bias severity increases.

Citations

• DiffuseMix: Label-Preserving Data Augmentation with Diffusion Models
• Dream the Impossible: Outlier Imagination with Diffusion Models
• Towards Last-layer Retraining for Group Robustness with Fewer Annotations
• Fill-Up: Balancing Long-Tailed Data with Generative Models
• Synthetic Data from Diffusion Models Improves ImageNet Classification
• Effective Data Augmentation With Diffusion Models
• Avoiding spurious correlations via logit correction
• Bias Mimicking: A Simple Sampling Approach for Bias Mitigation
• DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation
• An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion
• Discover and Mitigate Unknown Biases with Debiasing Alternate Networks
• Contrastive Adapters for Foundation Model Group Robustness
• Hierarchical Text-Conditional Image Generation with CLIP Latents
• Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations
• Fairness-aware Adversarial Perturbation Towards Bias Mitigation for Deployed Deep Models
• Correct-N-Contrast: A Contrastive Approach for Improving Robustness to Spurious Correlations
• Learning Fair Classifiers with Partially Annotated Group Labels
• Are Gender-Neutral Queries Really Gender-Neutral? Mitigating Gender Bias in Image Search
• Generative Adversarial Networks
• Unsupervised Learning of Debiased Representations with Pseudo-Attributes
• Just Train Twice: Improving Group Robustness without Training Group\n Information
• Learning Debiased Representation via Disentangled Feature Augmentation
• Learning Transferable Visual Models From Natural Language Supervision
• Fair Attribute Classification through Latent Space De-biasing
• Latent Adversarial Debiasing: Mitigating Collider Bias in Deep Neural Networks
• Environment Inference for Invariant Learning
• Denoising Diffusion Implicit Models
• Learning from Failure: Training Debiased Classifier from Biased Classifier
• Self-Challenging Improves Cross-Domain Generalization
• Fairness without Demographics through Adversarially Reweighted Learning
• Contrastive Examples for Addressing the Tyranny of the Majority
• Towards Fairness in Visual Recognition: Effective Strategies for Bias Mitigation
• Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization
• REPAIR: Removing Representation Bias by Dataset Resampling
• Learning Not to Learn: Training Deep Neural Networks with Biased Data
• BAGAN: Data Augmentation with Balancing GAN
• Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints
• Age Progression/Regression by Conditional Adversarial Autoencoder
• Deep Residual Learning for Image Recognition
• From Fake to Real: Pretraining on Balanced Synthetic Images to Prevent Spurious Correlations in Image Recognition

Related

• FairGen: Enhancing Fairness in Text-to-Image Diffusion Models via Self-Discovering Latent Directions
• Debiasing Diffusion Model: Enhancing Fairness through Latent Representation Learning in Stable Diffusion Model
• Invariant Learning via Diffusion Dreamed Distribution Shifts
• Balancing Act: Distribution-Guided Debiasing in Diffusion Models
• Training on Thin Air: Improve Image Classification with Generated Data