Pretraining on noisy, internet-scale datasets has been heavily studied as a technique for training models with broad, general capabilities for text, images, and other modalities. However, for many sequential decision domains such as robotics, video games, and computer use, publicly available data does not contain the labels required to train behavioral priors in the same way. We extend the internet-scale pretraining paradigm to sequential decision domains through semi-supervised imitation learning wherein agents learn to act by watching online unlabeled videos. Specifically, we show that with a small amount of labeled data we can train an inverse dynamics model accurate enough to label a huge unlabeled source of online data -- here, online videos of people playing Minecraft -- from which we can then train a general behavioral prior. Despite using the native human interface (mouse and keyboard at 20Hz), we show that this behavioral prior has nontrivial zero-shot capabilities and that it can be fine-tuned, with both imitation learning and reinforcement learning, to hard-exploration tasks that are impossible to learn from scratch via reinforcement learning. For many tasks our models exhibit human-level performance, and we are the first to report computer agents that can craft diamond tools, which can take proficient humans upwards of 20 minutes (24,000 environment actions) of gameplay to accomplish.

One of the biggest challenges in cyber-physical system (CPS) design is their intrinsic complexity, heterogeneity, and multidisciplinary nature. Emerging distributed CPSs integrate a wide range of heterogeneous aspects such as physical dynamics, control, machine learning, and error handling. Furthermore, system components are often distributed over multiple physical locations, hardware platforms, and communication networks. While model-based design (MBD) has tremendously improved the design process, CPS design remains a difficult task. Models are meant to improve understanding of a system, yet this quality is often lost when models become too complicated. In this paper, we show how to use aspect-oriented (AO) modeling techniques in MBD as a systematic way to segregate domains of expertise and cross-cutting concerns within the model. We demonstrate these concepts on actor-oriented models of an industrial robotic swarm application and illustrate the use of AO modeling techniques to manage the complexity. We also show how to use AO modeling for design-space exploration.

Accuracy and responsiveness are two key properties of emerging cyber-physical energy systems that need to incorporate high throughput sensor streams for distributed monitoring and control applications. The electric power grid, which is a prominent example of such systems, is being integrated with high throughput sensors in order to support stable system dynamics that are provisioned to be utilized in real-time supervisory control applications. The end-to-end performance and overall scalability of cyber-physical energy applications depend on robust middleware services that are able to operate with variable resources and multi-source sensor data. This leads to uncertain behavior under highly variable sensor and middleware topologies. We present a parametric approach to modeling the middleware service architecture for distributed power applications and account for temporal satisfiability of system properties under network resource and data volume uncertainty. We present a heterogeneous modeling framework that combines Monte Carlo simulations of uncertainty parameters within an executable discrete-event middleware service model. By employing Monte Carlo simulations followed by regression analysis, we quantify system parameters that significantly affect behavior of middleware services and the achievability of temporal requirements.

System simulation is a valuable tool to unveil inefficiencies and to test new strategies when implementing and revising systems. Often, simulations are parameterized using offline data and heuristic knowledge. Operational data, i.e., data gained through experimentation and observation, can greatly improve the fidelity between the actual system and the simulation. In a traffic scenario, for example, different road conditions or vehicle types can impact the outcome of the simulation and have to be considered during the modeling stage. This paper proposes using machine learning techniques to generate high fidelity simulation models. A traffic simulation case study exemplifies this approach by generating a model for the SUMO traffic simulator from vehicular telemetry data.