2024/05/10 by Jianyu Zhang, Zhang, Jianyu, Niklas Nolte +7 · 3 voices · 2 citations
Computer Science · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Neural and Evolutionary Computing (cs.NE) #cs.AI #cs.LG #cs.NE
paper · pdf · doi:10.48550/arxiv.2405.06394
arxiv published 2024/05/10 · arxiv updated 2025/02/27
Memory Mosaics are networks of associative memories working in concert to achieve a prediction task of interest. Like transformers, memory mosaics possess compositional capabilities and in-context learning capabilities. Unlike transformers, memory mosaics achieve these capabilities in comparatively transparent way ("predictive disentanglement"). We illustrate these capabilities on a toy example and also show that memory mosaics perform as well or better than transformers on medium-scale language modeling tasks.