DiffInk: Glyph- and Style-Aware Latent Diffusion Transformer for Text to Online Handwriting Generation

2025/09/28 by Pan, Wei, He, Huiguo, Cheng, Hiuyi +2
#Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2509.23624

Abstract

Deep generative models have advanced text-to-online handwriting generation (TOHG), which aims to synthesize realistic pen trajectories conditioned on textual input and style references. However, most existing methods still primarily focus on character- or word-level generation, resulting in inefficiency and a lack of holistic structural modeling when applied to full text lines. To address these issues, we propose DiffInk, the first latent diffusion Transformer framework for full-line handwriting generation. We first introduce InkVAE, a novel sequential variational autoencoder enhanced with two complementary latent-space regularization losses: (1) an OCR-based loss enforcing glyph-level accuracy, and (2) a style-classification loss preserving writing style. This dual regularization yields a semantically structured latent space where character content and writer styles are effectively disentangled. We then introduce InkDiT, a novel latent diffusion Transformer that integrates target text and reference styles to generate coherent pen trajectories. Experimental results demonstrate that DiffInk outperforms existing state-of-the-art methods in both glyph accuracy and style fidelity, while significantly improving generation efficiency. Code will be made publicly available.

Citations

• TrInk: Ink Generation with Transformer Network
• XVerse: Consistent Multi-Subject Control of Identity and Semantic Attributes via DiT Modulation
• REPA-E: Unlocking VAE for End-to-End Tuning with Latent Diffusion Transformers
• Zero-Shot Styled Text Image Generation, but Make It Autoregressive
• LeanVAE: An Ultra-Efficient Reconstruction VAE for Video Diffusion Models
• Compression-Aware One-Step Diffusion Model for JPEG Artifact Removal
• Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models
• OminiControl: Minimal and Universal Control for Diffusion Transformer
• Decoupling Layout from Glyph in Online Chinese Handwriting Generation
• DeepSeek LLM: Scaling Open-Source Language Models with Longtermism
• HierSpeech++: Bridging the Gap between Semantic and Acoustic Representation of Speech by Hierarchical Variational Inference for Zero-shot Speech Synthesis
• Few shot font generation via transferring similarity guided global style and quantization local style
• Zero-shot Generation of Training Data with Denoising Diffusion Probabilistic Model for Handwritten Chinese Character Recognition
• Handwritten Text Generation from Visual Archetypes
• Disentangling Writer and Character Styles for Handwriting Generation
• CF-Font: Content Fusion for Few-shot Font Generation
• GPT-4 Technical Report
• Adding Conditional Control to Text-to-Image Diffusion Models
• Adding Conditional Control to Text-to-Image Diffusion Models
• ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders
• Scalable Diffusion Models with Transformers
• Semantic-Conditional Diffusion Networks for Image Captioning
• Complex Handwriting Trajectory Recovery: Evaluation Metrics and Algorithm
• Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding
• Hierarchical Text-Conditional Image Generation with CLIP Latents
• SLOGAN: Handwriting Style Synthesis for Arbitrary-Length and Out-of-Vocabulary Text
• High-Resolution Image Synthesis with Latent Diffusion Models
• High-Resolution Image Synthesis with Latent Diffusion Models
• Content and Style Aware Generation of Text-Line Images for Handwriting Recognition
• Denoising Diffusion Implicit Models
• Generating Handwriting via Decoupled Style Descriptors
• wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
• Denoising Diffusion Probabilistic Models
• GANwriting: Content-Conditioned Generation of Styled Handwritten Word Images
• Understanding LSTM -- a tutorial into Long Short-Term Memory Recurrent Neural Networks
• DeepWriting: Making Digital Ink Editable via Deep Generative Modeling
• Neural Discrete Representation Learning
• Attention Is All You Need
• A Neural Representation of Sketch Drawings
• Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization
• Drawing and Recognizing Chinese Characters with Recurrent Neural Network
• U-Net: Convolutional Networks for Biomedical Image Segmentation
• Generating Sequences With Recurrent Neural Networks
• ALGORITHMS FOR THE REDUCTION OF THE NUMBER OF POINTS REQUIRED TO REPRESENT A DIGITIZED LINE OR ITS CARICATURE
• MegaTTS 3: Sparse Alignment Enhanced Latent Diffusion Transformer for Zero-Shot Speech Synthesis

Related

• DiffusionPen: Towards Controlling the Style of Handwritten Text Generation
• Diffusion models for Handwriting Generation
• Zero-Shot Paragraph-level Handwriting Imitation with Latent Diffusion Models
• ScriptViT: Vision Transformer-Based Personalized Handwriting Generation
• SLOGAN: Handwriting Style Synthesis for Arbitrary-Length and Out-of-Vocabulary Text