It is extremely difficult to train a superhuman Artificial Intelligence (AI) for games of similar size to StarCraft II. AlphaStar is the first AI that beat human professionals in the full game of StarCraft II, using a league training framework that is inspired by a game-theoretic approach. In this paper, we improve AlphaStar's league training in two significant aspects. We train goal-conditioned exploiters, whose abilities of spotting weaknesses in the main agent and the entire league are greatly improved compared to the unconditioned exploiters in AlphaStar. In addition, we endow the agents in the league with the new ability of opponent modeling, which makes the agent more responsive to the opponent's real-time strategy. Based on these improvements, we train a better and superhuman AI with orders of magnitude less resources than AlphaStar (see Table 1 for a full comparison). Considering the iconic role of StarCraft II in game AI research, we believe our method and results on StarCraft II provide valuable design principles on how one would utilize the general league training framework for obtaining a least-exploitable strategy in various, large-scale, real-world games.

We study the reinforcement learning problem of complex action control in the Multi-player Online Battle Arena (MOBA) 1v1 games. This problem involves far more complicated state and action spaces than those of traditional 1v1 games, such as Go and Atari series, which makes it very difficult to search any policies with human-level performance. In this paper, we present a deep reinforcement learning framework to tackle this problem from the perspectives of both system and algorithm. Our system is of low coupling and high scalability, which enables efficient explorations at large scale. Our algorithm includes several novel strategies, including control dependency decoupling, action mask, target attention, and dual-clip PPO, with which our proposed actor-critic network can be effectively trained in our system. Tested on the MOBA game Honor of Kings, the trained AI agents can defeat top professional human players in full 1v1 games.

MOBA games, e.g., Honor of Kings, League of Legends, and Dota 2, pose grand challenges to AI systems such as multi-agent, enormous state-action space, complex action control, etc. Developing AI for playing MOBA games has raised much attention accordingly. However, existing work falls short in handling the raw game complexity caused by the explosion of agent combinations, i.e., lineups, when expanding the hero pool in case that OpenAI's Dota AI limits the play to a pool of only 17 heroes. As a result, full MOBA games without restrictions are far from being mastered by any existing AI system. In this paper, we propose a MOBA AI learning paradigm that methodologically enables playing full MOBA games with deep reinforcement learning. Specifically, we develop a combination of novel and existing learning techniques, including off-policy adaption, multi-head value estimation, curriculum self-play learning, policy distillation, and Monte-Carlo tree-search, in training and playing a large pool of heroes, meanwhile addressing the scalability issue skillfully. Tested on Honor of Kings, a popular MOBA game, we show how to build superhuman AI agents that can defeat top esports players. The superiority of our AI is demonstrated by the first large-scale performance test of MOBA AI agent in the literature.

Programming by demonstration (PBD) transforms the robot programming from the code level to automated interface between robot and human, promoting the flexibility of robotized automation. In this paper, we focus on programming the industrial robot for assembly tasks by parsing the human demonstration into a series of assembly skills and compiling the skill to the robot executables. To achieve this goal, an identification system using multimodal information to recognize the assembly skill, called MASD, is proposed including: 1) an initial learning stage using a hierarchical model to recognize the action by considering the features from action-object effect, gesture, and trajectory and 2) a retrospective thinking stage using a segmentation method to cut the continuous demonstrations into multiple assembly skills optimally. Using MASD, the demonstration of assembly tasks can be explained with high accuracy in real time, driving a hypothesis that a PBD system on the top of MASD can be extended to more realistic assembly tasks beyond pure positional moving and picking. In experiments, the skill identification module is used to recognize the five kinds of assembly skills in demonstrations of both single and multiple assembly skills, and outperforms the comparative action identification methods. Besides integrated with the MASD, the PBD system can generate the program based on the demonstration and successfully enable an ABB industrial robotic arm simulator to assemble a flashlight and a switch, verifying the initial hypothesis. Note to Practitioners-In the conventional robotized automation, the key role of the robot mainly owes to its capacity for repeating a wide variety of tasks with high speed and accuracy in long term, with a cost of days to months of programming for deployment. On the other hand, the new trend of customization brings the new characteristics: production in short cycle and small volume. This irreversible momentum urges the robot to switch from task to task efficiently. The biggest bottleneck here is the tedious programming, which also has high prerequisites for most practitioners in manufacturing. This situation motivates the development of a PBD system that can understand the assembly skills performed by the human experts in the demonstration and accordingly generate the program for robot's execution of the taught task. In this paper, we present a skill decoding system to parse the observational raw demonstration into symbolic sequences, which is the crucial bridge to enable the automatic programming. The system achieves high performance in recognition and is tailored for the PBD in assembly tasks by considering both advantages and disadvantages in the background of assembly, such as controllable environment and limited computational resources. It is particularly useful for assembly tasks with modularized actions based on a set of standard parts. At the perspective of industrial application, the PBD upon the proposed system is a promising solution to improve the flexibility of manufacture, which is expected to be true in midterm but an important step toward this goal.

In this paper we propose a method for monocular visual-inertial odometry that utilizes image edges as measurements. In contrast to previous feature-based approaches, the proposed method does not employ any assumption on the geometry of the scene (e.g., it does not assume straight lines). It can thus use measurements from all image areas with significant gradient, similarly to direct semi-dense methods. However, in contrast to direct semi-dense approaches, the proposed method's measurement model is invariant to linear changes in the image intensity. The novel edge parameterization and measurement model we propose explicitly account for the fact that edge points can only provide useful information in the direction of the image gradient. We present both Monte-Carlo simulations, as well as results from real-world experimental testing, which demonstrate that the proposed edge-based approach to visual-inertial odometry is consistent, and outperforms the point-based one.

The vast majority of existing methods for vision-aided inertial navigation rely on the detection and tracking of point features in the images. However, in several man-made environments, such as indoor office spaces, straight line features are prevalent, while point features may be sparse. Therefore, developing methods that will enable the use of straight-line features for vision-aided inertial navigation can lead to improved performance. While limited prior work on the subject exists, it assumes the use of a global-shutter camera, i.e., a camera in which all image pixels are captured simultaneously. Most low-cost cameras, however, use rolling-shutter (RS) image capture, which renders the existing methods inapplicable. To address these limitations, we here present an algorithm for vision-aided inertial navigation that employs both point and line features, and is capable of operation with RS cameras. The two key contributions of this work are (i) a novel parameterization for 3D lines, which is shown to exhibit better linearity properties than existing ones, and (ii) a novel approach for the use of line observations in images. This approach forgoes line-fitting and does not assume that a straight line in 3D projects to a straight line in the image, and is thus suitable for use with RS cameras. Our results demonstrate that our proposed estimator formulation leads to improved precision, in point-feature-poor environments.

In this paper, we propose a high-precision pose estimation algorithm for systems equipped with low-cost inertial sensors and rolling-shutter cameras. The key characteristic of the proposed method is that it performs online self-calibration of the camera and the IMU, using detailed models for both sensors and for their relative configuration. Specifically, the estimated parameters include the camera intrinsics (focal length, principal point, and lens distortion), the readout time of the rolling-shutter sensor, the IMU's biases, scale factors, axis misalignment, and g-sensitivity, the spatial configuration between the camera and IMU, as well as the time offset between the timestamps of the camera and IMU. An additional contribution of this work is a novel method for processing the measurements of the rolling-shutter camera, which employs an approximate representation of the estimation errors, instead of the state itself. We demonstrate, in both simulation tests and real-world experiments, that the proposed approach is able to accurately calibrate all the considered parameters in real time, and leads to significantly improved estimation precision compared to existing approaches.