Achieving human-like planning and control with multimodal observations in an open world is a key milestone for more functional generalist agents. Existing approaches can handle certain long-horizon tasks in an open world. However, they still struggle when the number of open-world tasks could potentially be infinite and lack the capability to progressively enhance task completion as game time progresses. We introduceJARVIS-1, an open-world agent that can perceive multimodal input (visual observations and human instructions), generate sophisticated plans, and perform embodied control, all within the popular yet challenging open-world Minecraft universe. Specifically, we developJARVIS-1 on top of pre-trained multimodal language models, which map visual observations and textual instructions to plans. The plans will be ultimately dispatched to the goal-conditioned controllers. We outfitJARVIS-1 with a multimodal memory, which facilitates planning using both pre-trained knowledge and its actual game survival experiences.JARVIS-1 is the existing most general agent in Minecraft, capable of completing over 200 different tasks using control and observation space similar to humans. These tasks range from short-horizon tasks, e.g., “chopping trees” to long-horizon ones, e.g., “obtaining a diamond pickaxe”.JARVIS-1 performs exceptionally well in short-horizon tasks, achieving nearly perfect performance. In the classic long-term task ofObtainDiamondPickaxe,JARVIS-1 surpasses the reliability of current state-of-the-art agents by 5 times and can successfully complete longer-horizon and more challenging tasks. Furthermore, we show thatJARVIS-1 is able toself-improvefollowing a life-long learning paradigm thanks to multimodal memory, sparking a more general intelligence and improved autonomy.

This paper proposes a robust multi-image based blind face hallucination framework to super-resolve LR faces. The proposed framework first estimates both blurring kernel and transformations of multiple LR faces by robust deblurring and registration in PCA subspace. A patch-wise mixture of probabilistic PCA prior is then incorporated for face super-resolution. Previous work on face SR using PCA prior can be viewed as special cases of the framework. Experimental results in both simulated and real LR sequences demonstrate very promising performance of the proposed method.

The paper presents a joint framework for face hallucination incorporating face deblurring and registration. The joint framework not only directly hallucinates low resolution faces, but also deblurs and aligns low resolution faces iteratively to improve the performance of face hallucination. Without the need for accurate face registration and prior knowledge of blurring kernels, it is robust to errors in face registration and blurring kernel. Experimental results demonstrate the robust performance of the proposed method.

The paper presents a probabilistic tracking framework to fuse high-level object detection cues with low-level image feature cues using particle filters. First, an adaptive ICONDENSATION (AICONDENSATION) is introduced to better exploit object detection cues to guide importance sampling, where the proposal distribution is derived in a more principled approach using data association methods so that mixture weights can be adapted dynamically rather than fixed in ICONDENSATION. An adaptive detection fusion CONDENSATION (AFCONDENSATION) is further presented to directly fuse high-level object detection cues with low-level cues, where mixture weights are also adapted and it is shown that weight correction in ICONDENSATION actually is not necessary. Results on sequences with both simulated and real detections show improved performance of AI/AFCONDENSATION in comparison with ICONDENSATION.

The paper presents a fuzzy chamfer distance and its probabilistic formulation for edge-based visual tracking. First, connections of the chamfer distance and the Hausdorff distance with fuzzy objective functions for clustering are shown using a reformulation theorem. A fuzzy chamfer distance (FCD) based on fuzzy objective functions and a probabilistic formulation of the fuzzy chamfer distance (PFCD) based on data association methods are then presented for tracking, which can all be regarded as reformulated fuzzy objective functions and minimized with iterative algorithms. Results on challenging sequences demonstrate the performance of the proposed tracking method.

The paper proposes an edge-based multi-object tracking framework which deals with tracking multiple objects with occlusions using a variational particle filter. Object is modelled by a mixture of a non-parametric contour model and a non-parametric edge model using kernel density estimation. Visual tracking with a mixture model is formulated as a Bayesian incomplete data problem, where measurements in an image are associated with a generative model which is a mixture of mixture models including object models and a clutter model and unobservable associations of measurements to densities in the generative model are regarded as missing data. A likelihood for tracking multiple objects jointly with an exclusion principle is presented, where it is assumed that one measurement can only be generated from one density and one density can generate multiple measurements and it significantly reduces the complexity of enumerating all feasible events. To address the curse of dimensionality in tracking multiple objects jointly, a variational particle filter (VPF) is proposed for multi-object tracking, where the proposal distribution is based on the approximated posterior from variational inference rather than using the prior as the proposal distribution in sampling importance resampling (SIR) particle filter. With the variational particle filter, the number of particles needed for multi-object tracking can be significantly reduced. Experimental results in challenging sequences demonstrate the robust performance of the proposed method.

The paper presents a robust multi-feature head tracker using importance sampling particle filter with automatic head detection for video surveillance applications. Automatic head detection is based on moving region contour analysis using constraints of head shape and curvature scale space corner detector is introduced to segment contours. Due to the automatic head detection, we propose to exploit detected head shape cue to guide the sampling using importance sampling particle filter and individualized colour and edge features are fused for measurement. Experimental results demonstrate the robustness of the proposed head tracker.

This paper presents an automatic motion trajectory based surveillance video retrieval system with an efficient and flexible trajectory matching technique based on turning function. Raw trajectories of moving objects are extracted by automatic tracking and described by MPEG-7 motion trajectory descriptor with first order interpolating function. Efficient matching is performed for translation invariant and translation-rotation invariant video retrieval in spatial model and spatio-temporal model. In spatial model, turning function is introduced to represent the trail of motion. In spatio-temporal model, velocity function is used to represent the motion of object. Results of experiment on a prototype video database indicate a promising performance of the system. 1