Training diffusion models for audiovisual sequences allows for a range of generation tasks by learning conditional distributions of various input-output combinations of the two modalities. Nevertheless, this strategy often requires training a separate model for each task which is expensive. Here, we propose a novel training approach to effectively learn arbitrary conditional distributions in the audiovisual space. Our key contribution lies in how we parameterize the diffusion timestep in the forward diffusion process. Instead of the standard fixed diffusion timestep, we propose applying variable diffusion timesteps across the temporal dimension and across modalities of the inputs. This formulation offers flexibility to introduce variable noise levels for various portions of the input, hence the term mixture of noise levels. We propose a transformer-based audiovisual latent diffusion model and show that it can be trained in a task-agnostic fashion using our approach to enable a variety of audiovisual generation tasks at inference time. Experiments demonstrate the versatility of our method in tackling cross-modal and multimodal interpolation tasks in the audiovisual space. Notably, our proposed approach surpasses baselines in generating temporally and perceptually consistent samples conditioned on the input. Project page: neurips13025.github.io

Embedded computing devices play an integral role in the mechanical operations of modern-day vehicles. These devices exchange information containing critical vehicle parameters that reflect the current state of operations. Such information can be captured for various purposes, such as diagnostics, fleet management, and analytics. Although monitoring individual parameters can be useful for some applications, monitoring distinct combinations of parameters can reveal more complex and higher-level states that may give useful information. Existing monitoring systems either lack user configurability and control or present simple user interfaces that make it difficult to monitor and collate different parameters to observe high-level vehicle states. In this work, we present TruckSTM, a novel application that realizes user-defined states from messages seen in the embedded networks of medium and heavy duty vehicles and displays state transitions on an interactive user interface. We begin by symbolically formulating some of the in-vehicle networking concepts and formally defining the concept of operational states and state transitions. We then elaborate on the operations performed by TruckSTM in mapping network-obtained vehicle parameters to states that can be defined in standard JSON format. Finally, we evaluate TruckSTM’s asymptotic performance and present the results for the worst-case scenario and demonstrate that in a real world scenario such high level state visualization constraints of an operational truck.

Vehicle security has been receiving a lot of attention from both the black hat and white hat community of late. Research in this area has already led to the fabrication of different attacks, of which some have been shown to have potentially grave consequences. Vehicle vendors and original equipment manufacturers (OEM)s are thus presented with the additional responsibility of ensuring in-vehicular communication level security. In this poster paper, we present a framework, which allows any individual to write, test, and store exploit scripts which could then be run by any interested party on in-vehicular networks of commercial vehicles like trucks and buses.

Current approaches to video forecasting attempt to generate videos directly in pixel space using Generative Adversarial Networks (GANs) or Variational Autoencoders (VAEs). However, since these approaches try to model all the structure and scene dynamics at once, in unconstrained settings they often generate uninterpretable results. Our insight is that forecasting needs to be done first at a higher level of abstraction. Specifically, we exploit human pose detectors as a free source of supervision and break the video forecasting problem into two discrete steps. First we explicitly model the high level structure of active objects in the scene (humans) and use a VAE to model the possible future movements of humans in the pose space. We then use the future poses generated as conditional information to a GAN to predict the future frames of the video in pixel space. By using the structured space of pose as an intermediate representation, we sidestep the problems that GANs have in generating video pixels directly. We show through quantitative and qualitative evaluation that our method outperforms state-of-the-art methods for video prediction.

Vehicles now include Electronic Control Units (ECUs) that communicate with each other via broadcast networks. Cyber-security professionals have shown that such embedded communication networks can be compromised. Very recently, it has been shown that embedded devices connected to commercial vehicle networks can be manipulated to perform unintended actions by injecting spoofed messages. Such attacks can be hard to detect as they can mimic safety critical actions performed by ECUs. We present a precedence graph-based anomaly detection technique to detect malicious message injections. Our approach can detect malicious message injections and is able to distinguish them from safety critical actions like hard braking.

Given a scene, what is going to move, and in what direction will it move? Such a question could be considered a non-semantic form of action prediction. In this work, we present a convolutional neural network (CNN) based approach for motion prediction. Given a static image, this CNN predicts the future motion of each and every pixel in the image in terms of optical flow. Our CNN model leverages the data in tens of thousands of realistic videos to train our model. Our method relies on absolutely no human labeling and is able to predict motion based on the context of the scene. Because our CNN model makes no assumptions about the underlying scene, it can predict future optical flow on a diverse set of scenarios. We outperform all previous approaches by large margins.

In this paper we present a conceptually simple but surprisingly powerful method for visual prediction which combines the effectiveness of mid-level visual elements with temporal modeling. Our framework can be learned in a completely unsupervised manner from a large collection of videos. However, more importantly, because our approach models the prediction framework on these mid-level elements, we can not only predict the possible motion in the scene but also predict visual appearances - how are appearances going to change with time. This yields a visual "hallucination" of probable events on top of the scene. We show that our method is able to accurately predict and visualize simple future events, we also show that our approach is comparable to supervised methods for event prediction.