Mixture of Raytraced Experts

2025/07/16 by Andrea Perin, Giacomo Lagomarsini, Perin, Andrea +5
Social Sciences · #Artificial Intelligence (cs.AI) #Conferences and Exhibitions Management #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · pdf · doi:10.48550/arxiv.2507.12419

openalex publication_date 2025/07/16 · openalex created_date 2025/10/14 · openalex updated_date 2026/07/28

Abstract

We introduce a Mixture of Raytraced Experts, a stacked Mixture of Experts (MoE) architecture which can dynamically select sequences of experts, producing computational graphs of variable width and depth. Existing MoE architectures generally require a fixed amount of computation for a given sample. Our approach, in contrast, yields predictions with increasing accuracy as the computation cycles through the experts' sequence. We train our model by iteratively sampling from a set of candidate experts, unfolding the sequence akin to how Recurrent Neural Networks are trained. Our method does not require load-balancing mechanisms, and preliminary experiments show a reduction in training epochs of 10% to 40% with a comparable/higher accuracy. These results point to new research directions in the field of MoEs, allowing the design of potentially faster and more expressive models. The code is available at https://github.com/nutig/RayTracing

Citations

• Chain-of-Experts: Unlocking the Communication Power of Mixture-of-Experts Models
• Load Balancing Mixture of Experts with Similarity Preserving Routers
• MoE-GPS: Guidlines for Prediction Strategy for Dynamic Expert Duplication in MoE Load Balancing
• Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts
• Harder Tasks Need More Experts: Dynamic Routing in MoE Models
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
• Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
• BranchyNet: Fast Inference via Early Exiting from Deep Neural Networks

Related

• Multilinear Mixture of Experts: Scalable Expert Specialization through Factorization
• SEER-MoE: Sparse Expert Efficiency through Regularization for Mixture-of-Experts
• Elastic MoE: Unlocking the Inference-Time Scalability of Mixture-of-Experts
• Ray-Tracing for Conditionally Activated Neural Networks
• A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applications