Transformer-based Nonlinear Equalization for DP-16QAM Coherent Optical Communication Systems

2023/08/23 by Naveenta Gautam, Sai Vikranth Pendem, Gautam, Naveenta +5
Engineering · #Advanced Photonic Communication Systems #FOS: Electrical engineering #Optical Network Technologies #Semiconductor Lasers and Optical Devices #Signal Processing (eess.SP) #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2308.12262

openalex publication_date 2023/08/23 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Compensating for nonlinear effects using digital signal processing (DSP) is complex and computationally expensive in long-haul optical communication systems due to intractable interactions between Kerr nonlinearity, chromatic dispersion (CD), and amplified spontaneous emission (ASE) noise from inline amplifiers. The application of machine learning architectures has demonstrated promising advancements in enhancing transmission performance through the mitigation of fiber nonlinear effects. In this paper, we apply a Transformer-based model to dual-polarisation (DP)-16QAM coherent optical communication systems. We test the performance of the proposed model for different values of fiber lengths and launched optical powers and show improved performance compared to the state-of-the-art digital backpropagation (DBP) algorithm, fully connected neural network (FCNN) and bidirectional long short term memory (BiLSTM) architecture.

Related

• Compensation of Fiber Nonlinearities in Digital Coherent Systems Leveraging Long Short-Term Memory Neural Networks
• Complexity Reduction over Bi-RNN-Based Nonlinearity Mitigation in Dual-Pol Fiber-Optic Communications via a CRNN-Based Approach
• Recent Advances on Machine Learning-aided DSP for Short-reach and Long-haul Optical Communications
• Equalization in Dispersion-Managed Systems Using Learned Digital Back-Propagation
• Neural Turbo Equalization: Deep Learning for Fiber-Optic Nonlinearity\n Compensation