Exploring RWKV for Memory Efficient and Low Latency Streaming ASR

2023/09/26 by Keyu An, Shiliang Zhang, An, Keyu +1 · 1 citation
Computer Science · Earth and Planetary Sciences · #Audio and Speech Processing (eess.AS) #FOS: Computer and information sciences #FOS: Electrical engineering #Sound (cs.SD) #Speech Recognition and Synthesis #Speech and Audio Processing #Underwater Acoustics Research #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2309.14758

openalex publication_date 2023/09/26 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Recently, self-attention-based transformers and conformers have been introduced as alternatives to RNNs for ASR acoustic modeling. Nevertheless, the full-sequence attention mechanism is non-streamable and computationally expensive, thus requiring modifications, such as chunking and caching, for efficient streaming ASR. In this paper, we propose to apply RWKV, a variant of linear attention transformer, to streaming ASR. RWKV combines the superior performance of transformers and the inference efficiency of RNNs, which is well-suited for streaming ASR scenarios where the budget for latency and memory is restricted. Experiments on varying scales (100h - 10000h) demonstrate that RWKV-Transducer and RWKV-Boundary-Aware-Transducer achieve comparable to or even better accuracy compared with chunk conformer transducer, with minimal latency and inference memory cost.

Cited by

• Accurate, fast, cheap: Choose three. Replacing Multi-Head-Attention with Bidirectional Recurrent Attention for Long-Form ASR

Related

• CUSIDE-T: Chunking, Simulating Future and Decoding for Transducer based Streaming ASR
• Streaming Transformer-based Acoustic Models Using Self-attention with Augmented Memory
• Transformer-based Streaming ASR with Cumulative Attention
• A Better and Faster End-to-End Model for Streaming ASR
• Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Loss