2025/08/26 by Tatiana Likhomanenko, Richard He Bai, Ren-Yuan Bai +19 · 2 citations
Computer Science · Engineering · #Architecture #Dialog box #Dialog system #Language model #Language understanding #Latency (audio) #Low latency (capital markets) #Speech Recognition and Synthesis #Speech and dialogue systems #Speech processing #Topic Modeling #cs.CL #cs.LG #cs.SD #eess.AS
paper · pdf · doi:10.48550/arxiv.2509.00078
published in arXiv (Cornell University) (Cornell University)
openalex publication_date 2025/08/26 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05
The emergence of large language models (LLMs) has transformed spoken dialog systems, yet the optimal architecture for real-time on-device voice agents remains an open question. While end-to-end approaches promise theoretical advantages, cascaded systems (CSs) continue to outperform them in language understanding tasks, despite being constrained by sequential processing latency. In this work, we introduce ChipChat, a novel low-latency CS that overcomes traditional bottlenecks through architectural innovations and streaming optimizations. Our system integrates streaming (a) conversational speech recognition with mixture-of-experts, (b) state-action augmented LLM, (c) text-to-speech synthesis, (d) neural vocoder, and (e) speaker modeling. Implemented using MLX, ChipChat achieves sub-second response latency on a Mac Studio without dedicated GPUs, while preserving user privacy through complete on-device processing. Our work shows that strategically redesigned CSs can overcome their historical latency limitations, offering a promising path forward for practical voice-based AI agents.