Composed Fine-Tuning: Freezing Pre-Trained Denoising Autoencoders for\n Improved Generalization

2020/06/29 by Sang Michael Xie, Tengyu Ma, Xie, Sang Michael +3
Computer Science · #Adversarial Robustness in Machine Learning #Domain Adaptation and Few-Shot Learning #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Data Classification #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2006.16205

openalex publication_date 2020/06/29 · openalex created_date 2021/06/22 · openalex updated_date 2026/07/28

Abstract

We focus on prediction problems with structured outputs that are subject to\noutput validity constraints, e.g. pseudocode-to-code translation where the code\nmust compile. While labeled input-output pairs are expensive to obtain,\n"unlabeled" outputs, i.e. outputs without corresponding inputs, are freely\navailable (e.g. code on GitHub) and provide information about output validity.\nWe can capture the output structure by pre-training a denoiser to denoise\ncorrupted versions of unlabeled outputs. We first show that standard\nfine-tuning after pre-training destroys some of this structure. We then propose\ncomposed fine-tuning, which fine-tunes a predictor composed with the\npre-trained denoiser, which is frozen to preserve output structure. For\ntwo-layer ReLU networks, we prove that composed fine-tuning significantly\nreduces the complexity of the predictor, thus improving generalization.\nEmpirically, we show that composed fine-tuning improves over standard\nfine-tuning on two pseudocode-to-code translation datasets (3% and 6%\nrelative). The improvement from composed fine-tuning is magnified on\nout-of-distribution (OOD) examples (4% and 25% relative).\n

Citations

• Connectionist models of recognition memory: Constraints imposed by learning and forgetting functions.
• Sequence to Sequence Learning with Neural Networks
• Actor-Mimic: Deep Multitask and Transfer Reinforcement Learning
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• Categorical Reparameterization with Gumbel-Softmax
• Parameter-Efficient Transfer Learning for NLP
• Improved protein structure prediction using potentials from deep learning
• Prefix-Tuning: Optimizing Continuous Prompts for Generation

Related

• Towards Efficient Fine-tuning of Pre-trained Code Models: An Experimental Study and Beyond
• Scheduled denoising autoencoders
• How fine can fine-tuning be? Learning efficient language models
• Improving Compositionality of Neural Networks by Decoding Representations to Inputs
• Mind the Gap: Data Rewriting for Stable Off-Policy Supervised Fine-Tuning