One of the main challenges of multimodal learning is combining multiple heterogeneous modalities, e.g., video, audio, and text. Video and audio are obtained at much higher rates than text and are roughly aligned in time. They are often not synchronized with text, which comes as a global context, e.g. a title, or a description. Furthermore, video and audio inputs are of much larger volumes, and grow as the video length increases, which naturally requires more compute dedicated to these modalities, and makes modeling of long-range dependencies harder. We here decouple the multimodal modeling, dividing it into separate autoregressive models, processing the inputs according to the characteristics of the modalities. We propose a multimodal model, consisting of an autoregressive component for the time-synchronized modalities (audio and video), and an autoregressive component for the context modalities which are not necessarily aligned in time but are still sequential. To address the long-sequences of the video-audio inputs, we further partition the video and audio sequences in consecutive snippets and autoregressively process their representations. To that end, we propose a Combiner mechanism, which models the audio-video information jointly, producing compact but expressive representations. This allows us to scale to 512 input video frames without increase in model parameters. Our approach achieves the state-of-the-art on multiple well established multimodal benchmarks. It effectively addresses the high computational demand of media inputs by learning compact representations, controlling the sequence length of the audio-video feature representations, and modeling their dependencies in time.

Head-mounted displays (HMDs) in virtual reality (VR) occlude the upper face of the wearing users, leading to decreased nonverbal communication cues toward outside users. In this paper, we discuss Googly Eyes, a high-fidelity prototype that displays an illustration of the HMD-wearing user’s eyes in real-time in front of an HMD. We explored the effects of the Googly Eyes on the user experience of non-HMD users. For this, we designed and developed a collaborative asymmetrical co-located task performed by an HMD-wearing user and a non-HMD (tablet) user, and we conducted a between-subjects user study where we compared the Googly Eyes with a baseline HMD experience without any external eye depiction. In this paper, we discuss the system, task, user study details, and results along with implications for future studies.