Abstract Group activity recognition in video is a complex task due to the need for a model to recognise the actions of all individuals in the video and their complex interactions. Recent studies propose that optimal performance is achieved by individually tracking each person and subsequently inputting the sequence of poses or cropped images/optical flow into a model. This helps the model to recognise what actions each person is performing before they are merged to arrive at the group action class. However, all previous models rely highly on high-quality tracking and have only been evaluated using ground truth tracking information. In practice, achieving highly reliable tracking information for all individuals in a group activity video is almost impossible. We introduce an innovative deep learning-based group activity recognition approach called Rendered Pose-based Group Activity Recognition System (RePGARS), designed to tolerate unreliable tracking and pose information. Experimental results confirm that RePGARS outperforms all existing group activity recognition algorithms tested, which do not use ground truth detection and tracking information.

Abstract We introduce a novel deep learning-based group activity recognition approach called the Pose Only Group Activity Recognition System (POGARS), designed to use only tracked poses of people to predict the performed group activity. In contrast to existing approaches for group activity recognition, POGARS uses 1D CNNs to learn spatiotemporal dynamics of individuals involved in a group activity and forgo learning features from pixel data. The proposed model uses a spatial and temporal attention mechanism to infer person-wise importance and multi-task learning for simultaneously performing group and individual action classification. Experimental results confirm that POGARS achieves highly competitive results compared to state-of-the-art methods on a widely used public volleyball dataset despite only using tracked pose as input. Further, our experiments show by using pose only as input, POGARS has better generalization capabilities compared to methods that use RGB as input.

Automatically determining three-dimensional human pose from monocular RGB image data is a challenging problem. The two-dimensional nature of the input results in intrinsic ambiguities which make inferring depth particularly difficult. Recently, researchers have demonstrated that the flexible statistical modelling capabilities of deep neural networks are sufficient to make such inferences with reasonable accuracy. However, many of these models use coordinate output techniques which are memory-intensive, not differentiable, and/or do not spatially generalise well. We propose improvements to 3D coordinate prediction which avoid the aforementioned undesirable traits by predicting 2D marginal heatmaps under an augmented soft-argmax scheme. Our resulting model, MargiPose, produces visually coherent heatmaps whilst maintaining differentiability. We are also able to achieve state-of-the-art accuracy on publicly available 3D human pose estimation data.

Camera calibration is a preliminary step in sports analytics which enables us to transform player positions to standard playing area coordinates. While many camera calibration systems work well when the visual content contains sufficient clues, such as a key frame, calibrating without such information, such as may be needed when processing footage captured by a coach from the sidelines or stands, is challenging. In this paper an innovative automatic camera calibration system, which does not make use of any key frames, is presented for sports analytics. The proposed system consists of three components: a robust linear panorama module, a playing area estimation module, and a homography estimation module. It can eliminate distortion and calibrate the camera in each frame simultaneously, using correspondences between pairs of consecutive frames. Experiments on real data evaluate the performance and demonstrate the robustness of the system.

In professional sport, an enormous amount of fine-grain performance data can be generated at near millisecond intervals in the form of vision-based tracking data. One of the first sports to embrace this technology has been tennis, where Hawk-Eye technology has been used to both aid umpiring decisions, and to visualize shot trajectories for broadcast purposes. Despite the high-level of accuracy of the tracking systems and the sheer volume of spatiotemporal data they generate, the use of this data for player performance analysis and prediction has been lacking. In this research, we use ball and player tracking data from “Hawk-Eye” to discover unique player styles and predict within-point events. We move beyond current analysis that only incorporates coarse match statistics (i.e., serves, winners, number of shots, and volleys) and use spatial and temporal information which better characterizes the tactics and tendencies of each player. Using a probabilistic graphical model, we are able to model player behaviors which enables us to: 1) find the factors such as location and speed of the incoming shot which are most conducive to a player hitting a winner (i.e., “sweet-spot”) or cause an error, and 2) do “live in-point” prediction - based on the shots being played during a rally we estimate the probability of the outcome (e.g., winner, continuation, or error) and the location of the next shot. As player behavior depends on the opponent, we use model adaptation to enhance our prediction. We show the utility of our approach by analyzing the play of Djokovic, Nadal, and Federer at the 2012 Australian Tennis Open.

In competitive combat sporting environments like boxing, the statistics on a boxer's performance, including the amount and type of punches thrown, provide a valuable source of data and feedback which is routinely used for coaching and performance improvement purposes. This paper presents a robust framework for the automatic Classification of a boxer's punches. Overhead depth imagery is employed to alleviate challenges associated with occlusions, and robust body-part tracking is developed for the noisy time-of-flight sensors. Punch recognition is addressed through both a multi-class SVM and Random Forest classifiers. coarse-to-fine hierarchical SVM classifier is presented based on prior knowledge of boxing punches. This framework has been applied to shadow boxing image sequences taken at the Australian Institute of Sport with 8 elite boxers. Results demonstrate the effectiveness of the proposed approach, with the hierarchical SVM classifier yielding a 96% accuracy, signifying its suitability for analysing athletes punches in boxing bouts.

In professional sport, an enormous amount of fine-grain performance data can be generated at near millisecond intervals in the form of vision-based tracking data. One of the first sports to embrace this technology has been tennis, where Hawk-Eye technology has been used to both aid umpiring decisions, and to visualize shot trajectories for broadcast purposes. These data have tremendous untapped applications in terms of "opponent planning'', where a large amount of recent data is used to learn contextual behavior patterns of individual players, and ultimately predict the likelihood of a particular type of serve. Since the type of serve selected by a player may be contingent on the match context (i.e., is the player down break-point, or is serving for the match etc.), the characteristics of the player (i.e., the player may have a very fast serve, hit heavy with topspin or kick, or slice serves into the body) as well as the characteristics of the opponent (e.g., the opponent may prefer to play from the baseline or "chip-and-charge'' into the net). In this paper we present a method which recommends the most likely serves of a player in a given context. We show by utilizing a "style prior", we can improve the prediction/recommendation. Such an approach also allows us to quantify the similarity between players, which is useful in enriching the dataset for future prediction. We conduct our analysis on Hawk-Eye data collected from three recent Australian Open Grand-Slam Tournaments and show how our approach can be used in practice.

In elite sports, nearly all performances are captured on video. Despite the massive amounts of video that has been captured in this domain over the last 10-15 years, most of it remains in an “unstructured” or “raw” form, meaning it can only be viewed or manually annotated/tagged with higher-level event labels which is time consuming and subjective. As such, depending on the detail or depth of annotation, the value of the collected repositories of archived data is minimal as it does not lend itself to large-scale analysis and retrieval. One such example is swimming, where each race of a swimmer is captured on a camcorder and in-addition to the split-times (i.e., the time it takes for each lap), stroke rate and stroke-lengths are manually annotated. In this paper, we propose a vision-based system which effectively “digitizes” a large collection of archived swimming races by estimating the location of the swimmer in each frame, as well as detecting the stroke rate. As the videos are captured from moving hand-held cameras which are located at different positions and angles, we show our hierarchical-based approach to tracking the swimmer and their different parts is robust to these issues and allows us to accurately estimate the swimmer location and stroke rates.

In this paper, we describe a method to represent and discover adversarial group behavior in a continuous domain. In comparison to other types of behavior, adversarial behavior is heavily structured as the location of a player (or agent) is dependent both on their teammates and adversaries, in addition to the tactics or strategies of the team. We present a method which can exploit this relationship through the use of a spatiotemporal basis model. As players constantly change roles during a match, we show that employing a "role-based" representation instead of one based on player "identity" can best exploit the playing structure. As vision-based systems currently do not provide perfect detection/tracking (e.g. missed or false detections), we show that our compact representation can effectively "denoise" erroneous detections as well as enabling temporal analysis, which was previously prohibitive due to the dimensionality of the signal. To evaluate our approach, we used a fully instrumented field-hockey pitch with 8 fixed high-definition (HD) cameras and evaluated our approach on approximately 200,000 frames of data from a state-of-the-art real-time player detector and compare it to manually labelled data.