For certain applications like highway surveillance systems, only low-frame-rate videos are recorded, which presents a huge challenge to existing trackers, as objects tend to undergo far more abrupt changes in location, motion, and appearance between successive frames compared to normal frame rates. To handle the above challenges, we propose a novel approach, namely$\mathbb {SORT}$-$\mathbb {LFR}$, for$\mathbb {S}$imple$\mathbb {O}$nline and$\mathbb {R}$ealtime$\mathbb {T}$racking in$\mathbb {L}$ow-$\mathbb {F}$rame-$\mathbb {R}$ate videos, which consists of following techniques: 1) A feature-prior association strategy to improve the capability to track new objects with significant displacements; 2) A Kalman filter using acceleration in state space (accel-fused Kalman filter) to improve the motion estimation capability for non-constant velocity moving objects; 3) A detection-guided adaptive exponential moving average (DG-AEMA) feature update mechanism to enhance feature temporal modeling capability for tracked objects; 4) A trajectory-covariance threshold tuning (TCTT) method to filter out incorrect association results. Through these techniques, the proposed SORT achieves 91.8 HOTA, 92.6 MOTA and 93.9 IDF1, which surpass all state-of-the-art trackers on the public CityFlow and our private HighwayTrack datasets under the low-frame-rate setting.

Deep models for facial expression recognition achieve high performance by training on large-scale labeled data. However, publicly available datasets contain uncertain facial expressions caused by ambiguous annotations or confusing emotions, which could severely decline the robustness. Previous studies usually follow the bias elimination method in general tasks without considering the uncertainty problem from the perspective of different corresponding sources. This article proposes a novel method of multi-task assisted correction in addressing uncertain facial expression recognition called MTAC. Specifically, a confidence estimation block and a weighted regularization module are applied to highlight solid samples and suppress uncertain samples in every batch. In addition, two auxiliary tasks, i.e., action unit detection and valence-arousal measurement, are introduced to learn semantic distributions from a data-driven AU graph and mitigate category imbalance based on latent dependencies between discrete and continuous emotions, respectively. Moreover, a re-labeling strategy guided by feature-level similarity constraint further generates new labels for identified uncertain samples to promote model learning. The proposed method can flexibly combine with existing frameworks in a fully-supervised or weakly-supervised manner. Experiments on five popular benchmarks demonstrate that the MTAC substantially improves over baselines when facing synthetic and real uncertainties and outperforms the state-of-the-art methods.

As one of the most important affective signals, facial affect analysis (FAA) is essential for developing human-computer interaction systems. Early methods focus on extracting appearance and geometry features associated with human affects while ignoring the latent semantic information among individual facial changes, leading to limited performance and generalization. Recent work attempts to establish a graph-based representation to model these semantic relationships and develop frameworks to leverage them for various FAA tasks. This paper provides a comprehensive review of graph-based FAA, including the evolution of algorithms and their applications. First, the FAA background knowledge is introduced, especially on the role of the graph. We then discuss approaches widely used for graph-based affective representation in literature and show a trend towards graph construction. For the relational reasoning in graph-based FAA, existing studies are categorized according to their non-deep or deep learning methods, emphasizing the latest graph neural networks. Performance comparisons of the state-of-the-art graph-based FAA methods are also summarized. Finally, we discuss the challenges and potential directions. As far as we know, this is the first survey of graph-based FAA methods. Our findings can serve as a reference for future research in this field.

High-quality annotated images are significant to deep facial expression recognition (FER) methods. However, uncertain labels, mostly existing in large-scale public datasets, often mislead the training process. In this paper, we achieve uncertain label correction of facial expressions using auxiliary action unit (AU) graphs, called ULC-AG. Specifically, a weighted regularization module is introduced to highlight valid samples and suppress category imbalance in every batch. Based on the latent dependency between emotions and AUs, an auxiliary branch using graph convolutional layers is added to extract the semantic information from graph topologies. Finally, a re-labeling strategy corrects the ambiguous annotations by comparing their feature similarities with semantic templates. Experiments show that our ULC-AG achieves 89.31% and 61.57% accuracy on RAF-DB and AffectNet datasets, respectively, outperform the baseline and state-of-the-art methods.

Facial expression plays an important role in human communication as a type of nonverbal language and has been widely used in various areas such as psychology, human-computer interaction and robotics. Nowadays, convolutional neural network is a promising approach for facial expression recognition. However, convolutional layers can be time-consuming and computationally expensive because a large number of parameters participate in the calculations and need to be updated during training. To improve the performance of deep neural network in facial expression recognition and accelerate training and calculation, we propose a novel framework which adopts efficient element-wise multiplication to replace traditional convolution. To disentangle reliable feature representation for more effective recognition and further enhance the recognition performance while maintaining the efficiency, we propose a representation scheme which can retain informative feature components while removing unreliable ones in Fourier domain based on the proposed multiplication framework. Extensive comparison and ablation studies are conducted on several benchmark datasets, which shows the efficiency and effectiveness of the proposed model.

The outbreak of coronavirus disease (COVID-19) has been a nightmare to citizens, hospitals, healthcare practitioners, and the economy in 2020. The overwhelming number of confirmed cases and suspected cases put forward an unprecedented challenge to the hospital's capacity of management and medical resource distribution. To reduce the possibility of cross-infection and attend a patient according to his severity level, expertly diagnosis and sophisticated medical examinations are often required but hard to fulfil during a pandemic. To facilitate the assessment of a patient's severity, this paper proposes a multi-modality feature learning and fusion model for end-to-end covid patient severity prediction using the blood test supported electronic medical record (EMR) and chest computerized tomography (CT) scan images. To evaluate a patient's severity by the co-occurrence of salient clinical features, the High-order Factorization Network (HoFN) is proposed to learn the impact of a set of clinical features without tedious feature engineering. On the other hand, an attention-based deep convolutional neural network (CNN) using pre-trained parameters are used to process the lung CT images. Finally, to achieve cohesion of cross-modality representation, we design a loss function to shift deep features of both-modality into the same feature space which improves the model's performance and robustness when one modality is absent. Experimental results demonstrate that the proposed multi-modality feature learning and fusion model achieves high performance in an authentic scenario.

Facial expression recognition has become a newly-emerging topic in recent decades, which has important value in the field of human-computer interaction. In this paper, we present a deep learning based approach, named frequency neural network (FreNet), for facial expression recognition. Different from convolutional neural network in spatial domain, FreNet inherits the advantages of processing image in frequency domain, such as efficient computation and spatial redundancy elimination. First, we propose the learnable multiplication kernel and construct multiple multiplication layers to learn features in frequency domain. Second, a summarization layer is proposed following multiplication layers to further yield high-level features. Third, based on the property of discrete cosine transform (DCT), we utilize multiplication layers and summarization layer to construct the Basic-FreNet, which can yield high-level features on the widely used DCT feature. Finally, to further achieve better performance on Basic-FreNet, we propose the Block-FreNet in which the weight-shared multiplication kernel is designed for feature learning and the block sub-sampling is designed for dimension reduction. The experimental results show that the Block-FreNet not only achieves superior performance, but also greatly reduces the computational cost. To our best knowledge, the proposed approach is the first attempt to fill in the blank of frequency based deep learning model for facial expression recognition.

Motions of facial components convey significant information of facial expressions. Although remarkable advancement has been made, the dynamic of facial topology has not been fully exploited. In this paper, a novel facial expression recognition (FER) algorithm called Spatial Temporal Semantic Graph Network (STSGN) is proposed to automatically learn spatial and temporal patterns through end-to-end feature learning from facial topology structure. The proposed algorithm not only has greater discriminative power to capture the dynamic patterns of facial expression and stronger generalization capability to handle different variations but also higher interpretability. Experimental evaluation on two popular datasets, CK+ and Oulu-CASIA, shows that our algorithm achieves more competitive results than other state-of-the-art methods.

Previous studies recognizing expressions with facial graph topology mostly use a fixed facial graph structure established by the physical dependencies among facial landmarks. However, the static graph structure inherently lacks flexibility in non-standardized scenarios. This paper proposes a dynamic-graph-based method for effective and robust facial expression recognition. To capture action-specific dependencies among facial components, we introduce a link inference structure, called the Situational Link Generation Module (SLGM). We further propose the Situational Graph Convolution Network (SGCN) to automatically detect and recognize facial expression in various conditions. Experimental evaluations on two lab-constrained datasets, CK+ and Oulu, along with an in-the-wild dataset, AFEW, show the superior performance of the proposed method. Additional experiments on occluded facial images further demonstrate the robustness of our strategy.

As implicit feedback can be tracked automatically and is easy to collect, the implicit recommendation attracts more researcher attentions. However, the uncertainty of the implicit feedback meaning poses a great challenge to the implicit recommendation. Especially for the missing data, we are not sure whether the users dislike or just have not seen the items. It may lead to bias of predictions. In this paper, we propose Neural Fuzzy Inference based on User preference and Item popularity (UI-NFI) algorithm to model the missing data in implicit recommendation. First, we use fuzzy set theory to represent user preference and item popularity that get from the history interactions and side information. Furthermore, neural fuzzy inference is proposed to predict the exposure possibility of missing data. Based on the fuzzy inference model, UI-NFI and matrix factorization model perform joint learning to predict. Experimental results show that our model has better performance compared to the other implicit recommendation algorithms.

We explore how to leverage the performance of face feature points detection on mobile terminals from 3 aspects. First, we optimize the models used in SDM algorithms via PCA and Spectrum Clustering. Second, we propose an evaluation criterion using Linear Discriminative Analysis to choose the best local feature descriptions which plays a critical role in feature points detection. Third, we take advantage of multicore architecture of mobile terminal and parallelize the optimized SDM algorithm to improve the efficiency further. The experiment observations show that our final accomplished GPC‐SDM (improved Supervised Descent Method using spectrum clustering, PCA, and GPU acceleration) suppresses the memory usage, which is beneficial and efficient to meet the real‐time requirements.