Enforcing Encoder-Decoder Modularity in Sequence-to-Sequence Models

2019/11/09 by Dalmia, Siddharth, Mohamed, Abdelrahman, Lewis, Mike +2
#Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.1911.03782

Abstract

Inspired by modular software design principles of independence, interchangeability, and clarity of interface, we introduce a method for enforcing encoder-decoder modularity in seq2seq models without sacrificing the overall model quality or its full differentiability. We discretize the encoder output units into a predefined interpretable vocabulary space using the Connectionist Temporal Classification (CTC) loss. Our modular systems achieve near SOTA performance on the 300h Switchboard benchmark, with WER of 8.3% and 17.6% on the SWB and CH subsets, using seq2seq models with encoder and decoder modules which are independent and interchangeable.

Related

• LegoNN: Building Modular Encoder-Decoder Models
• Seq vs Seq: An Open Suite of Paired Encoders and Decoders
• Recipes for Sequential Pre-training of Multilingual Encoder and Seq2Seq Models
• Return of the Encoder: Maximizing Parameter Efficiency for SLMs
• Decoder-Only or Encoder-Decoder? Interpreting Language Model as a Regularized Encoder-Decoder