Mixtures of Experts Unlock Parameter Scaling for Deep RL

The recent rapid progress in (self) supervised learning models is in largepart predicted by empirical scaling laws: a model's performance scalesproportionally to its size. Analogous scaling laws remain elusive forreinforcement learning domains, however, where increasing the parameter countof a model often hurts its final performance. In this paper, we demonstratethat incorporating Mixture-of-Expert (MoE) modules, and in particular Soft MoEs(Puigcerver et al., 2023), into value-based networks results in moreparameter-scalable models, evidenced by substantial performance increasesacross a variety of training regimes and model sizes. This work thus providesstrong empirical evidence towards developing scaling laws for reinforcementlearning.