This paper describes an energy-based learning method that predicts the activities of multiple agents simultaneously. It aims to forecast both upcoming actions and paths of all agents in a scene based on their past activities, which can be jointly formulated by a probabilistic model over time. Learning this model is challenging because: 1) it has a large number of time-dependent variables that must scale with the forecast horizon and the number of agents; 2) distribution functions have to contain multiple modes in order to capture the spatio-temporal complexities of each agent's activities. To address these challenges, we put forth a novel Energy-based Learning approach for Multi-Agent activity forecasting (ELMA) to estimate this complex model via maximum log-likelihood estimation. Specifically, by sampling from a sequence of factorized marginalized multi-model distributions, ELMA generates most possible future actions efficiently. Moreover, by graph-based representations, ELMA also explicitly resolves the spatio-temporal dependencies of all agents' activities in a single pass. Our experiments on two large-scale datasets prove that ELMA outperforms recent leading studies by an obvious margin.

Multi-person action forecasting is an emerging task and a pivotal step towards video understanding. The major challenge lies in estimating a distribution characterizing the upcoming actions of all individuals in the scene. The state-of-the-art solutions attempt to solve this problem via a step-by-step prediction procedure. However, they are not adequate to address some particular limitations, such as the compounding errors, the innate uncertainty of the future and the spatio-temporal contexts. To handle the multi-person action forecasting challenges, we put forth a novel imitative learning framework upon the basis of inverse reinforcement learning. Specifically, we aim to learn a policy to model the aforementioned distribution up to a coming horizon through an objective that naturally solves the compounding errors. Such a policy is able to explore multiple plausible futures via extrapolating a series of latent variables and taking them into account to generate predictions. The impacts of these latent variables are further investigated by optimizing the directed information. Moreover, we reason the spatial context along with the temporal cue in a single pass with the usage of graph structural data. The experimental outcomes on two large-scale datasets reveal that our approach yields considerable improvements in terms of both diversity and quality with respect to recent leading studies.

How to deal with occlusion among persons plays important role in robust multi-pedestrian tracking field. This paper proposes to handle inter-person occlusion by introducing social interaction - attraction and grouping from 2D image plane. This social interaction is systematically studied based on spatial-temporal information without utilizing any scene knowledge. Attraction is integrated into hierarchical data-association as a penalizing factor, while grouping is used to aid target's motion dynamic. The comparison experimental results show that our method achieves results on par with, even better than state-of-the-art online tracking methods.