General movements (GMs) are part of the spontaneous movement repertoire and are present from early fetal life onwards up to age five months. GMs are connected to infants' neurological development and can be qualitatively assessed via the General Movement Assessment (GMA). In particular, between the age of three to five months, typically developing infants produce Fidgety Movements (FM) and their absence provides strong evidence for the presence of cerebral palsy (CP). To improve accessibility to the GMA, automated GMA solutions have been a key research area with proposed models becoming increasingly more accurate and interpretable. However, current models cannot gauge their ability to make decisions, which may lead to overconfident mistakes. To address this issue, we propose a Deep learning-based approach that not only classifies movements as fidgety or non-fidgety but also selectively abstains from classification when uncertain. Through two novel regularization losses, our model maintains a balanced coverage across the two movement types, which prevents bias toward an easy-to-classify subset of movements. We show that our proposed model learns to gauge its own confidence on movement classification, and our proposed regularization losses effectively ensure that the model maintains a similar confidence across movement types. We also show that the local movement abstentions have little impact on the video-level coverage and that relying on the most confident predictions improves the video-level performance.

Typically developing infants, between the corrected age of 9-20 weeks, produce fidgety movements. These movements can be identified with the General Movement Assessment, but their identification requires trained professionals to conduct the assessment from video recordings. Since trained professionals are expensive and their demand may be higher than their availability, computer vision-based solutions have been developed to assist practitioners. However, most solutions to date treat the problem as a direct mapping from video to infant status, without modeling fidgety movements throughout the video. To address that, we propose to directly model infants' short movements and classify them as fidgety or non-fidgety. In this way, we model the explanatory factor behind the infant's status and improve model interpretability. The issue with our proposal is that labels for an infant's short movements are not available, which precludes us to train such a model. We overcome this issue with active learning. Active learning is a framework that minimizes the amount of labeled data required to train a model, by only labeling examples that are considered "informative" to the model. The assumption is that a model trained on informative examples reaches a higher performance level than a model trained with randomly selected examples. We validate our framework by modeling the movements of infants' hips on two representative cohorts: typically developing and at-risk infants. Our results show that active learning is suitable to our problem and that it works adequately even when the models are trained with labels provided by a novice annotator.

Fidgety movements occur in infants between the age of 9 to 20 weeks post-term, and their absence are a strong indicator that an infant has cerebral palsy. Prechtl's General Movement Assessment method evaluates whether an infant has fidgety movements, but requires a trained expert to conduct it. Timely evaluation facilitates early interventions, and thus computer-based methods have been developed to aid domain experts. However, current solutions rely on complex models or high-dimensional representations of the data, which hinder their interpretability and generalization ability. To address that we propose [Formula: see text], a method that detects fidgety movements and uses them towards an assessment of the quality of an infant's general movements. [Formula: see text] is true to the domain expert process, more accurate, and highly interpretable due to its fine-grained scoring system. The main idea behind [Formula: see text] is to specify signal properties of fidgety movements that are measurable and quantifiable. In particular, we measure the movement direction variability of joints of interest, for movements of small amplitude in short video segments. [Formula: see text] also comprises a strategy to reduce those measurements to a single score that quantifies the quality of an infant's general movements; the strategy is a direct translation of the qualitative procedure domain experts use to assess infants. This brings [Formula: see text] closer to the process a domain expert applies to decide whether an infant produced enough fidgety movements. We evaluated [Formula: see text] on the largest clinical dataset reported, where it showed to be interpretable and more accurate than many methods published to date.

Human activities can be learned from video. With effective modeling it is possible to discover not only the action labels but also the temporal structure of the activities, such as the progression of the sub-activities. Automatically recognizing such structure from raw video signal is a new capability that promises authentic modeling and successful recognition of human-object interactions. Toward this goal, we introduce Asynchronous-Sparse Interaction Graph Networks (ASSIGN), a recurrent graph network that is able to automatically detect the structure of interaction events associated with entities in a video scene. ASSIGN pioneers learning of autonomous behavior of video entities including their dynamic structure and their interaction with the coexisting neighbors. Entities’ lives in our model are asynchronous to those of others therefore more flexible in adapting to complex scenarios. Their interactions are sparse in time hence more faithful to the true underlying nature and more robust in inference and learning. ASSIGN is tested on human-object interaction recognition and shows superior performance in segmenting and labeling of human sub-activities and object affordances from raw videos. The native ability of ASSIGN in discovering temporal structure also eliminates the dependence on external segmentation that was previously mandatory for this task.

Appearance features have been widely used in video anomaly detection even though they contain complex entangled factors. We propose a new method to model the normal patterns of human movements in surveillance video for anomaly detection using dynamic skeleton features. We decompose the skeletal movements into two sub-components: global body movement and local body posture. We model the dynamics and interaction of the coupled features in our novel Message-Passing Encoder-Decoder Recurrent Network. We observed that the decoupled features collaboratively interact in our spatio-temporal model to accurately identify human-related irregular events from surveillance video sequences. Compared to traditional appearance-based models, our method achieves superior outlier detection performance. Our model also offers “open-box” examination and decision explanation made possible by the semantically understandable features and a network architecture supporting interpretability.

Imbalanced datasets are pervasive and comprise many real-world applications, such as medical diagnosis and software fault detection. As common classifiers assume a balanced distribution of examples in the data, learning from imbalanced datasets presents its own challenges. Sampling techniques play an essential role in aiding classifiers which learn from imbalanced datasets, as these techniques return a more balanced version of the imbalanced dataset. Given the current number of sampling techniques available, selecting a technique together with a set of values for its hyper-parameters is a time-consuming task. In this work, we treat the mentioned problem as a many-objective optimization problem. An evolutionary algorithm was applied to select sampling algorithms and their parameters to imbalanced datasets considering multiple performance criteria. In the experiments, we compared the proposed method against the brute-force, the default (all sampling algorithms with their default hyper-parameters' values), and the random approaches. The experiments revealed that the proposal reached results comparable to those achieved by the brute-force approach, overcame the techniques with their default parameters most of the time, and surpassed the random search approach in the majority of the problems.