We presentCentury Maze,1 a simulated environment for cooperative navigation of groups numbering over 100 robots with primarily visual observations. The underlying task of large-scale object search is designed to push the scalability and efficiency of decentralized multi-robot systems relying on communication to overcome the limitations of separate units. To enable the training of robotic agents in large-scale visual embodied applications such asCentury Maze, we propose two reinforcement learning (RL) components that facilitate the emergence of effective communication:Differentiable Inter-Agent Belief Learning (DIABL), which provides a clear learning signal by optimizing the agents' beliefs about their goals and propagating gradients back over a differentiable communication channel to include the appropriate observers, andInformative Event Replay (InfER), which compensates for the scarcity of samples relevant toDIABLby maintaining a dedicated buffer for auxiliary learning. We demonstrate our combined method through an actor-critic neural network architecture encompassing a decentralized multi-agent system of variable size with individual goal-based belief modules. Results onCentury Mazeshow that our method is able to significantly improve performance through cooperation between over 100 agents against comparable baselines and maintains its effectiveness in realistic conditions.