We present Ego-Exo4D, a diverse, large-scale multimodal multiview video dataset and benchmark challenge. Ego-Exo4D centers around simultaneously-captured egocentric and exocentric video of skilled human activities (e.g., sports, music, dance, bike repair). 740 participants from 13 cities worldwide performed these activities in 123 different natural scene contexts, yielding long-form captures from 1 to 42 minutes each and 1,286 hours of video combined. The multimodal nature of the dataset is unprecedented: the video is accompanied by multichannel audio, eye gaze, 3D point clouds, camera poses, IMU, and multiple paired language descriptions—including a novel “expert commentary” done by coaches and teachers and tailored to the skilled-activity domain. To push the frontier of first-person video understanding of skilled human activity, we also present a suite of benchmark tasks and their annotations, including fine-grained activity understanding, proficiency estimation, cross-view translation, and 3D hand/body pose. All resources are open sourced to fuel new research in the community. https://ego-exo4d-data.org/

We present Ego-Exo4D, a diverse, large-scale multi-modal multiview video dataset and benchmark challenge. Ego-Exo4D centers around simultaneously-captured ego-centric and exocentric video of skilled human activities (e.g., sports, music, dance, bike repair). 740 participants from 13 cities worldwide performed these activities in 123 different natural scene contexts, yielding long-form captures from 1 to 42 minutes each and 1,286 hours of video combined. The multimodal nature of the dataset is un-precedented: the video is accompanied by multichannel audio, eye gaze, 3D point clouds, camera poses, IMU, and multiple paired language descriptions-including a novel “expert commentary” done by coaches and teachers and tailored to the skilled-activity domain. To push the frontier of first-person video understanding of skilled human activity, we also present a suite of benchmark tasks and their annotations, including fine-grained activity understanding, proficiency estimation, cross-view translation, and 3D hand/body pose. All resources are open sourced to fuel new research in the community.

Stereo rectification is widely considered “solved” due to the abundance of traditional approaches to perform recti-fication. However, autonomous vehicles and robots in-the-wild require constant re-calibration due to exposure to var-ious environmental factors, including vibration, and structural stress, when cameras are arranged in a wide-baseline configuration. Conventional rectification methods fail in these challenging scenarios: especially for larger vehicles, such as autonomous freight trucks and semi-trucks, the resulting incorrect rectification severely affects the quality of downstream tasks that use stereo/multi-view data. To tackle these challenges, we propose an online rectification approach that operates at real-time rates while achieving high accuracy. We propose a novel learning-based online cal-ibration approach that utilizes stereo correlation volumes built from a feature representation obtained from cross-image attention. Our model is trained to minimize vertical optical flow as proxy rectification constraint, and predicts the relative rotation between the stereo pair. The method is real-time and even outperforms conventional methods used for offline calibration, and substantially improves downstream stereo depth, post-rectification. We release two public datasets (https://light.princeton.edu/online-stereo-recification/), a synthetic and experimental wide baseline dataset, to foster further research.

The paper presents the basic ideas behind the use of software agent technology for distributed simulation and data assimilation. A software agent is an autonomous computer program that operates on behalf of someone or something. A mobile agent has the ability to migrate during execution from machine to machine in a heterogeneous network, while a stationary agent executes only on the system on which it began execution. To understand the role of agents in distributed simulation, note that simulations often operate on static datasets and data sources. Many simulations would produce more accurate results if they could access dynamically changing data from other sources, such as sensors or even other simulations. From the perspective of one simulation, other simulations are data resources, producing information possibly relevant to the past, present, or future of the system being modeled. Software agents allow dynamic linking between distributed simulations and efficient monitoring of and access to remote data resources. Specifically, they conserve bandwidth, provide custom operations without precompiling or preloading, and adapt to support disconnected operations. The paper describes the development of a software agent based framework for dynamically linking distributed simulations and other remote data resources. The framework called ABELS (Agent-Based Environment for Linking Simulations) allows independently designed simulations to communicate seamlessly with no a priori knowledge of the details of other simulations and data sources. We discuss our architecture and current implementation developed using the D'Agents mobile agent system.