Learning rational behaviors in open-world games like Minecraft remains to be challenging for Reinforcement Learning (RL) research due to the compound challenge of partial observability, high-dimensional visual perception and delayed reward. To address this, we propose JueWu-MC, a sample-efficient hierarchical RL approach equipped with representation learning and imitation learning to deal with perception and exploration. Specifically, our approach includes two levels of hierarchy, where the high-level controller learns a policy to control over options and the low-level workers learn to solve each sub-task. To boost the learning of sub-tasks, we propose a combination of techniques including 1) action-aware representation learning which captures underlying relations between action and representation, 2) discriminator-based self-imitation learning for efficient exploration, and 3) ensemble behavior cloning with consistency filtering for policy robustness. Extensive experiments show that JueWu-MC significantly improves sample efficiency and outperforms a set of baselines by a large margin. Notably, we won the championship of the NeurIPS MineRL 2021 research competition and achieved the highest performance score ever.

We are motivated by a recently developed nonlinear inverse scale space method for image denoising [M. Burger, G. Gilboa, S. Osher, and J. Xu, Commun. Math. Sci., 4 (2006), pp. 179–212; M. Burger, S. Osher, J. Xu, and G. Gilboa, in Variational, Geometric, and Level Set Methods in Computer Vision, Lecture Notes in Comput. Sci. 3752, Springer, Berlin, 2005, pp. 25–36], whereby noise can be removed with minimal degradation. The additive noise model has been studied extensively, using the Rudin–Osher–Fatemi model [L. I. Rudin, S. Osher, and E. Fatemi, Phys. D, 60 (1992), pp. 259–268], an iterative regularization method [S. Osher, M. Burger, D. Goldfarb, J. Xu, and W. Yin, Multiscale Model. Simul., 4 (2005), pp. 460–489], and the inverse scale space flow [M. Burger, G. Gilboa, S. Osher, and J. Xu, Commun. Math. Sci., 4 (2006), pp. 179–212; M. Burger, S. Osher, J. Xu, and G. Gilboa, in Variational, Geometric, and Level Set Methods in Computer Vision, Lecture Notes in Comput. Sci. 3752, Springer, Berlin, 2005, pp. 25–36]. However, the multiplicative noise model has not yet been studied thoroughly. Earlier total variation models for the multiplicative noise cannot easily be extended to the inverse scale space, due to the lack of global convexity. In this paper, we review existing multiplicative models and present a new total variation framework for the multiplicative noise model, which is globally strictly convex. We extend this convex model to the nonlinear inverse scale space flow and its corresponding relaxed inverse scale space flow. We demonstrate the convergence of the flow for the multiplicative noise model, as well as its regularization effect and its relation to the Bregman distance. We investigate the properties of the flow and study the dependence on flow parameters. The numerical results show an excellent denoising effect and significant improvement over earlier multiplicative models.