We present Self-Organizing Visual Prototypes (SOP), a new training technique for unsupervised visual feature learning. Unlike existing prototypical self-supervised learning (SSL) methods that rely on a single prototype to encode all relevant features of a hidden cluster in the data, we propose the SOP strategy. In this strategy, a prototype is represented by many semantically similar representations, or support embeddings (SEs), each containing a complementary set of features that together better characterize their region in space and maximize training performance. We reaffirm the feasibility of non-parametric SSL by introducing novel non-parametric adaptations of two loss functions that implement the SOP strategy. Notably, we introduce the SOP Masked Image Modeling (SOP-MIM) task, where masked representations are reconstructed from the perspective of multiple non-parametric local SEs. We comprehensively evaluate the representations learned using the SOP strategy on a range of benchmarks, including retrieval, linear evaluation, fine-tuning, and object detection. Our pre-trained encoders achieve state-of-the-art performance on many retrieval benchmarks and demonstrate increasing performance gains with more complex encoders.

Vision Transformers rely on fixed patch tokens that ignore the spatial and semantic structure of images. In this work, we introduce an end-to-end differentiable tokenizer that adapts to image content with pixel-level granularity while remaining backward-compatible with existing architectures for retrofitting pretrained models. Our method uses hierarchical model selection with information criteria to provide competitive performance in both image-level classification and dense-prediction tasks, and even supports out-of-the-box raster-to-vector conversion.

This paper introduces a novel approach to improving the training stability of self-supervised learning (SSL) methods by leveraging a non-parametric memory of seen concepts. The proposed method involves augmenting a neural network with a memory component to stochastically compare current image views with previously encountered concepts. Additionally, we introduce stochastic memory blocks to regularize training and enforce consistency between image views. We extensively benchmark our method on many vision tasks, such as linear probing, transfer learning, few-shot classification, and image retrieval on many datasets. The experimental results consolidate the effectiveness of the proposed approach in achieving stable SSL training without additional regularizers while learning highly transferable representations and requiring less computing time and resources.

We present Consistent Assignment of Views over Random Partitions (CARP), a self-supervised clustering method for representation learning of visual features. CARP learns prototypes in an end-to-end online fashion using gradient descent without additional non-differentiable modules to solve the cluster assignment problem. CARP optimizes a new pretext task based on random partitions of prototypes that regularizes the model and enforces consistency between views' assignments. Additionally, our method improves training stability and prevents collapsed solutions in joint-embedding training. Through an extensive evaluation, we demonstrate that CARP's representations are suitable for learning downstream tasks. We evaluate CARP's representations capabilities in 17 datasets across many standard protocols, including linear evaluation, few-shot classification, $k$-NN, $k$-means, image retrieval, and copy detection. We compare CARP performance to 11 existing self-supervised methods. We extensively ablate our method and demonstrate that our proposed random partition pretext task improves the quality of the learned representations by devising multiple random classification tasks. In transfer learning tasks, CARP achieves the best performance on average against many SSL methods trained for a longer time.

Facial expression recognition has been studied broadly, and several works using local micro-pattern descriptors have obtained significant results. There are, however, open questions: how to design a discriminative and robust feature descriptor?, how to select expression-related most influential features?, and how to represent the face descriptor exploiting the most salient parts of the face? In this article, we address these three issues to achieve better performance in recognizing facial expressions. First, we propose a new feature descriptor, namely Local Shape Pattern (LSP), that describes the local shape structure of a pixel’s neighborhood based on the prominent directional information by analyzing the statistics of the neighborhood gradient, which allows it to be robust against subtle local noise and distortion. Furthermore, we propose a selection strategy for learning the influential codes being active in the expression affiliated changes by selecting them exhibiting statistical dominance and high spatial variance. Lastly, we learn the size of the salient facial blocks to represent the facial description with the notion that changes in expressions vary in size and location. We conduct person-independent experiments in existing datasets after combining above three proposals, and obtain an improved performance for the facial expression recognition task.

Independent components within low-dimensional representations are essential inputs in several downstream tasks, and provide explanations over the observed data. Video-based disentangled factors of variation provide low-dimensional representations that can be identified and used to feed task-specific models. We introduce MTC-VAE, a self-supervised motion-transfer VAE model to disentangle motion and content from videos. Unlike previous work on video content-motion disentanglement, we adopt a chunk-wise modeling approach and take advantage of the motion information contained in spatiotemporal neighborhoods. Our model yields independent per-chunk representations that preserve temporal consistency. Hence, we reconstruct whole videos in a single forward-pass. We extend the ELBO's log-likelihood term and include a Blind Reenactment Loss as an inductive bias to leverage motion disentanglement, under the assumption that swapping motion features yields reenactment between two videos. We evaluate our model with recently-proposed disentanglement metrics and show that it outperforms a variety of methods for video motion-content disentanglement. Experiments on video reenactment show the effectiveness of our disentanglement in the input space where our model outperforms the baselines in reconstruction quality and motion alignment.

Anomaly detection suffers from unbalanced data since anomalies are quite rare. Synthetically generated anomalies are a solution to such ill or not fully defined data. However, synthesis requires an expressive representation to guarantee the quality of the generated data. In this article, we propose a two-level hierarchical latent space representation that distills inliers' feature descriptors [through autoencoders (AEs)] into more robust representations based on a variational family of distributions (through a variational AE) for zero-shot anomaly generation. From the learned latent distributions, we select those that lie on the outskirts of the training data as synthetic-outlier generators. Also, we synthesize from them, i.e., generate negative samples without seen them before, to train binary classifiers. We found that the use of the proposed hierarchical structure for feature distillation and fusion creates robust and general representations that allow us to synthesize pseudo outlier samples. Also, in turn, train robust binary classifiers for true outlier detection (without the need for actual outliers during training). We demonstrate the performance of our proposal on several benchmarks for anomaly detection.

Deep learning models need large amounts of data for training. In video recognition and classification, significant advances were achieved with the introduction of new large databases. However, the creation of large-databases for training is infeasible in several scenarios. Thus, existing or small collected databases are typically joined and amplified to train these models. Nevertheless, training neural networks on limited data is not straightforward and comes with a set of problems. In this paper, we explore the effects of stacking databases, model initialization, and data amplification techniques when training with limited data on deep learning models’ performance. We focused on the problem of Facial Expression Recognition from videos. We performed an extensive study with four databases at a different complexity and nine deep-learning architectures for video classification. We found that (i) complex training sets translate better to more stable test sets when trained with transfer learning and synthetically generated data, but their performance yields a high variance; (ii) training with more detailed data translates to more stable performance on novel scenarios (albeit with lower performance); (iii) merging heterogeneous data is not a straightforward improvement, as the type of augmentation and initialization is crucial; (iv) classical data augmentation cannot fill the holes created by joining largely separated datasets; and (v) inductive biases help to bridge the gap when paired with synthetic data, but this data is not enough when working with standard initialization techniques.

Automatic understanding of videos is one of the complex problems in machine learning and computer vision. An important area in the field of video analysis is human action recognition (HAR). Though a large number of HAR systems have already been developed, there is plenty of daily life actions that are difficult to recognize, due to several reasons, such as recording on different devices, poor video quality and similarities among actions. Development in the field of deep learning, especially in convolutional neural networks (CNN), has provided us with methods that are well-suited for the tasks of image and video recognition. This work implements a CNN-based hierarchical recognition approach to recognize 20 most difficult-to-recognize actions from the Kinetics dataset. Experimental results have shown that the application of our method significantly improves the quality of recognition for these actions.

In this paper, we propose Local Adaptive Hybrid Pattern (LAHP) to use intrinsic properties, both edge and color, of each pixel adaptively (inspired from LHP based methods) while using a single feature representation (inspired from LOBSTER). The proposed LAHP encodes edge and color information, and an adaptive factor based on gradient magnitude together as a single feature. We introduce a way to calculate the code distance of LNESP to reduce sensitive to edge distortions. Furthermore, we modify ADM (Adaptive Dictionary Model) to manage LAHP and extend a feature matching scheme of ADM to adjacent background models to reduce sensitivity to background motions. Results show promising performance than other methods in standard datasets.

Pulmonary nodules detection play a significant role in the early detection and treatment of lung cancer. False positive reduction is the one of the major parts of pulmonary nodules detection systems. In this study a novel method aimed at recognizing real pulmonary nodule among a large group of candidates was proposed. The method consists of three steps: appropriate receptive field selection, feature extraction and a strategy for high level feature fusion and classification. The dataset consists of 888 patient's chest volume low dose computer tomography (LDCT) scans, selected from publicly available LIDC-IDRI dataset. This dataset was marked by LUNA16 challenge organizers resulting in 1186 nodules. Trivial data augmentation and dropout were applied in order to avoid overfitting. Our method achieved high competition performance metric (CPM) of 0.735 and sensitivities of 78.8% and 83.9% at 1 and 4 false positives per scan, respectively. This study is also accompanied by detailed descriptions and results overview in comparison with the state of the art solutions.

Bone suppression in lung radiographs is an important task, as it improves the results on other related tasks, such as nodule detection or pathologies classification. In this paper, we propose two architectures that suppress bones in radiographs by treating them as noise. In the proposed methods, we create end-to-end learning frameworks that minimize noise in the images while maintaining sharpness and detail in them. Our results show that our proposed noise-cancellation scheme is robust and does not introduce artifacts into the images.

This paper presents a new face descriptor, local directional ternary pattern (LDTP), for facial expression recognition. LDTP efficiently encodes information of emotion-related features (ı.e., eyes, eyebrows, upper nose, and mouth) by using the directional information and ternary pattern in order to take advantage of the robustness of edge patterns in the edge region while overcoming weaknesses of edge-based methods in smooth regions. Our proposal, unlike existing histogram-based face description methods that divide the face into several regions and sample the codes uniformly, uses a two-level grid to construct the face descriptor while sampling expression-related information at different scales. We use a coarse grid for stable codes (highly related to non-expression), and a finer one for active codes (highly related to expression). This multi-level approach enables us to do a finer grain description of facial motions while still characterizing the coarse features of the expression. Moreover, we learn the active LDTP codes from the emotion-related facial regions. We tested our method by using person-dependent and independent cross-validation schemes to evaluate the performance. We show that our approaches improve the overall accuracy of facial expression recognition on six data sets.

In this paper, we propose a novel edge descriptor method for background modeling. In comparison to previous edge-based local-pattern methods, it is more robust to noise and illumination variations due to the use of principal gradient information in a local neighborhood. For the background modeling problem, we combined the proposed method with the Local Hybrid Pattern and experimented with an adaptive-dictionary-model based background modeling method. We show in the quantitative evaluations that the proposed methods is better than other local edge descriptors when applied to the same framework. Furthermore, we show that our proposed method is more powerful than other state of the art methods on standard datasets for the background modeling problem.

In this paper, we propose a hybrid background model that relies on edge and non-edge features of the image to produce the model. We encode these features into a coding scheme, that we called Local Hybrid Pattern (LHP), that selectively models edges and non-edges features of each pixel. Furthermore, we model each pixel with an adaptive code dictionary to represent the background dynamism, and update it by adding stable codes and discarding unstable ones. We weight each code in the dictionary to enhance its description of the pixel it models. The foreground is detected as the incoming codes that deviate from the dictionary. We can detect (as foreground or background) and classify (as edge or inner region) each pixel simultaneously. We tested our proposed method in existing databases with promising results.

Spatiotemporal image descriptors are gaining attention in the image research community for better representation of dynamic textures. In this paper, we introduce a dynamic-micro-texture descriptor, i.e., spatiotemporal directional number transitional graph (DNG), which describes both the spatial structure and motion of each local neighborhood by capturing the direction of natural flow in the temporal domain. We use the structure of the local neighborhood, given by its principal directions, and compute the transition of such directions between frames. Moreover, we present the statistics of the direction transitions in a transitional graph, which acts as a signature for a given spatiotemporal region in the dynamic texture. Furthermore, we create a sequence descriptor by dividing the spatiotemporal volume into several regions, computing a transitional graph for each of them, and represent the sequence as a set of graphs. Our results validate the robustness of the proposed descriptor in different scenarios for expression recognition and dynamic texture analysis.

Unattended object detection is an important task in surveillance. Thus, we propose a new method to detect unattended object by modeling the objects as newly learned temporal background. We use edge-segments to model the structural changes in the scene. Specifically, we construct distributions of these edge-segments to analyze the scene, and to segment its different components: background, fore-ground, and the interesting new objects. Additionally, we propose a clustering algorithm to recover the unattended objects from a set of edges based on the assumption that spatially close edges come from the same object. Our experiments on several datasets validate our proposed method.

For background-subtraction-based moving object detection, reliable background modeling is the most important component. Pixel-based methods are sensitive to illumination change, and edge-based methods can solve illumination-related problems, but have shape distortion problems. In this paper, we propose an edge-segment-based statistical background modeling algorithm and an online update mechanism to detect moving objects from consecutive frames, which creates a balance between the pixel- and edge-based methods. Our background modeling method uses a statistical map to model the frequency of the background-edges, as distributions that comprise support regions approximated with a quadratic function and enhanced with color and gradient information, to overcome the edge-distortion problem by matching the edge-segments to the modeled distributions. To adjust the changing background in the scenes, we propose an online-background update step for every incoming frame that updates the statistical map and enhances the information held by the distributions support regions. Furthermore, our experiments show that the proposed method obtains better results and detects moving edges efficiently.

Background modeling is challenging due to background dynamism. Most background modeling methods fail in the presence of intensity changes, because the model cannot handle sudden changes. A solution to this problem is to use intensity-robust features. Despite the changes of an edge's shape and position among frames, edges are less sensitive than a pixel's intensity to illumination changes. Furthermore, background models in the presence of moving objects produce ghosts in the detected output, because high quality models require ideal backgrounds. In this paper, we propose a robust statistical edge-segment-based method for background modeling of non-ideal sequences. The proposed method learns the structure of the scene using the edges' behaviors through the use of kernel-density distributions. Moreover, it uses segment features to overcome the shape and position variations of the edges. Hence, the use of segments gives us local information of the scene, and that helps us to predict the objects and background precisely. Furthermore, we accumulate segments to build edge distributions, which allow us to perform unconstrained training and to overcome the ghost effect. In addition, the proposed method uses adaptive thresholding (in the segments) to detect the moving objects. Therefore, this approach increases the accuracy over previous methods, which use fixed thresholds.

This paper proposes a novel local feature descriptor, local directional number pattern (LDN), for face analysis, i.e., face and expression recognition. LDN encodes the directional information of the face's textures (i.e., the texture's structure) in a compact way, producing a more discriminative code than current methods. We compute the structure of each micro-pattern with the aid of a compass mask that extracts directional information, and we encode such information using the prominent direction indices (directional numbers) and sign-which allows us to distinguish among similar structural patterns that have different intensity transitions. We divide the face into several regions, and extract the distribution of the LDN features from them. Then, we concatenate these features into a feature vector, and we use it as a face descriptor. We perform several experiments in which our descriptor performs consistently under illumination, noise, expression, and time lapse variations. Moreover, we test our descriptor with different masks to analyze its performance in different face analysis tasks.

In this paper, we propose a novel local feature descriptor, Local Sign Directional Pattern (LSDP), for face expression recognition. LSDP encodes the directional information of the face's textures—i.e., the texture's structure—in a compact way, producing a more discriminating code than other state-of-the-art methods. The structure of each micro-pattern is encoded by using its prominent directions and sign—which allows it to distinguish among similar structural patterns that have different intensity transitions. We divide the face into several regions, from which we extract the distributions of the LSDP features. These features are concatenated into a feature vector, and used as a face descriptor, and the expression recognition is obtained with the aid of Support Vector Machine classifiers.

Deriving an effective facial feature from original face images is a vital step for a successful automatic facial expression recognition. In this paper, we proposed a new feature descriptor, Local Principal Texture Pattern (LPTP), for expression recognition. We compute the LPTP feature, at each pixel, by extracting the principal directions of the local neighborhood, and we code the intensity differences on these directions. The mixture of direction and contrast information makes our descriptor robust against rotation and illumination changes. Consequently, we represent each expression as a distribution of LPTP codes. Our experiments demonstrate the superiority of the proposed feature, on two facial expression databases, over the existing methods.

The current contrast enhancement algorithms occasionally result in artifacts, overenhancement, and unnatural effects in the processed images. These drawbacks increase for images taken under poor illumination conditions. In this paper, we propose a content-aware algorithm that enhances dark images, sharpens edges, reveals details in textured regions, and preserves the smoothness of flat regions. The algorithm produces an ad hoc transformation for each image, adapting the mapping functions to each image's characteristics to produce the maximum enhancement. We analyze the contrast of the image in the boundary and textured regions, and group the information with common characteristics. These groups model the relations within the image, from which we extract the transformation functions. The results are then adaptively mixed, by considering the human vision system characteristics, to boost the details in the image. Results show that the algorithm can automatically process a wide range of images-e.g., mixed shadow and bright areas, outdoor and indoor lighting, and face images-without introducing artifacts, which is an improvement over many existing methods.

The detection of moving objects depends on the accuracy of the model used to represent the background. Common pixel-based and naive edge-based approaches have many drawbacks in dynamic environments, e.g., false detections with noise. We propose a novel background model that encodes the background as edges, building a statistical distribution per segment that represents the edge behavior. We build the background distributions using a kernel-based approach; the moving objects are detected as the edges that deviate from the distributions. The method does adaptive thresholding to the edges, which maintains their shape and boosts the detection accuracy. Sets of gradient distributions are incorporated into the model, to determine edges that lie within the distributions, but are moving edges. The number of distributions is handled dynamically, allowing them to increase and decrease accordingly to the situation. The experiments show that the proposed method improves the detection rates, due to its robustness against illumination changes.

We propose an edge segment based statistical backgroundmodeling algorithm and a moving edge detectionframework for the detection of moving objects. We analyzethe performance of the proposed segment based statisticalbackground model with traditional pixel based, edge pixelbased and edge segment based approaches. Existing edgebased moving object detection algorithms fetches difficultydue to the change in background motion, object shape, illuminationvariation and noise. The proposed algorithmmakes efficient use of statistical background model usingthe edge-segment structure. Experiments with natural imagesequences show that our method can detect moving objectsefficiently under the above mentioned environments.