We study the exploration-exploitation trade-off for large multiplayer coordination games where players strategise via Q-Learning, a common learning framework in multi-agent reinforcement learning. Q-Learning is known to have two shortcomings, namely non-convergence and potential equilibrium selection problems, when there are multiple fixed points, called Quantal Response Equilibria (QRE). Furthermore, whilst QRE have full support for finite games, it is not clear how Q-Learning behaves as the game becomes large. In this paper, we characterise the critical exploration rate that guarantees convergence to a unique fixed point, addressing the two shortcomings above. Using a generating-functional method, we show that this rate increases with the number of players and the alignment of their payoffs. For many-player coordination games with perfectly aligned payoffs, this exploration rate is roughly twice that of p-player zero-sum games. As for large games, we provide a structural result for QRE, which suggests that as the game size increases, Q-Learning converges to a QRE near the boundary of the simplex of the action space, a phenomenon we term asymptotic extinction, where a constant fraction of the actions are played with zero probability at a rate o(1/N) for an N -action game.

Seemingly minor details of mathematical and computational models of evolution are known to change the effect of population structure on the outcome of evolutionary processes. For example, birth-death dynamics often result in amplification of selection, while death-birth processes have been associated with suppression. In many biological populations the interaction structure is not static. Instead, members of the population are in motion and can interact with different individuals at different times. In this work we study populations embedded in a flowing medium; the interaction network is then time dependent. We use computer simulations to investigate how this dynamic structure affects the success of invading mutants, and compare these effects for different coupled birth and death processes. Specifically, we show how the speed of the motion impacts the fixation probability of an invading mutant. Flows of different speeds interpolate between evolutionary dynamics on fixed heterogeneous graphs and well-stirred populations; this allows us to systematically compare against known results for static structured populations. We find that motion has an active role in amplifying or suppressing selection by fragmenting and reconnecting the interaction graph. While increasing flow speeds suppress selection for most evolutionary models, we identify characteristic responses to flow for the different update rules we test. In particular we find that selection can be maximally enhanced or suppressed at intermediate flow speeds.