Variational graph auto-encoders (VGAEs) are a key tool for node clustering, but existing models face several significant challenges. These challenges include a mismatch between inference and generative models after incorporating the clustering inductive bias, as well as posterior collapse (PC), where latent representations become overly influenced by the prior distribution. In addition, in existing VGAEs, noisy clustering assignments lead to the feature randomness (FR) challenge, while the strong tradeoff between clustering accuracy and reconstruction quality results in the feature drift (FD) problem. To address these issues, we propose a multiscale contrastive VGAE (MCVGAE). This multiscale model combines cluster-level and graph-level contrastive learning with proximity-level and cluster-level self-supervised methods. MCVGAE improves the alignment between the hidden space and the data distribution and prevents PC. Moreover, it reduces FR and FD more effectively than existing techniques. Achieving impressive accuracy scores of 79.09% on Cora, 90.04% on ACM, 75.12% on Pubmed, 72.7% on Citeseer, 74.11% on DBLP, and 59.79% on Wiki clearly demonstrates the superiority of MCVGAE over 30 state-of-the-art methods.

Abstract In this work, we survey recent studies on masked image modeling (MIM), an approach that emerged as a powerful self-supervised learning technique in computer vision. The MIM task involves masking some information, e.g. pixels, patches, or even latent representations, and training a model, usually an autoencoder, to predicting the missing information by using the context available in the visible part of the input. We identify and formalize two categories of approaches on how to implement MIM as a pretext task, one based on reconstruction and one based on contrastive learning. Then, we construct a taxonomy and review the most prominent papers in recent years. We complement the manually constructed taxonomy with a dendrogram obtained by applying a hierarchical clustering algorithm. We further identify relevant clusters via manually inspecting the resulting dendrogram. Our review also includes datasets that are commonly used in MIM research. We aggregate the performance results of various masked image modeling methods on the most popular datasets, to facilitate the comparison of competing methods. Finally, we identify research gaps and propose several interesting directions of future work. We supplement our survey with the following public repository containing organized references: https://github.com/vladhondru25/MIM-Survey .

Most methods for conditional video synthesis use a single modality as the condition. This comes with major limitations. For example, it is problematic for a model conditioned on an image to generate a specific motion trajectory desired by the user since there is no means to provide motion information. Conversely, language information can describe the desired motion, while not precisely defining the content of the video. This work presents a multimodal video generation framework that benefits from text and images provided jointly or separately. We leverage the recent progress in quantized representations for videos and apply a bidirectional transformer with multiple modalities as inputs to predict a discrete video representation. To improve video quality and consistency, we propose a new video token trained with self-learning and an improved mask-prediction algorithm for sampling video tokens. We introduce text augmentation to improve the robustness of the textual representation and diversity of generated videos. Our framework can incorporate various visual modalities, such as segmentation masks, drawings, and partially occluded images. It can generate much longer sequences than the one used for training. In addition, our model can extract visual information as suggested by the text prompt, e.g., “an object in image one is moving northeast”, and generate corresponding videos. We run evaluations on three public datasets and a newly collected dataset labeled with facial attributes, achieving state-of-the-art generation results on all four11Code: https://github.com/snap-research/MMVID and Webpage..

Image segmentation is a key task in computer vision and image processing with important applications such as scene understanding, medical image analysis, robotic perception, video surveillance, augmented reality, and image compression, among others, and numerous segmentation algorithms are found in the literature. Against this backdrop, the broad success of deep learning (DL) has prompted the development of new image segmentation approaches leveraging DL models. We provide a comprehensive review of this recent literature, covering the spectrum of pioneering efforts in semantic and instance segmentation, including convolutional pixel-labeling networks, encoder-decoder architectures, multiscale and pyramid-based approaches, recurrent networks, visual attention models, and generative models in adversarial settings. We investigate the relationships, strengths, and challenges of these DL-based segmentation models, examine the widely used datasets, compare performances, and discuss promising research directions.

In this paper, we propose scalable methods for maximizing a regularized submodular function $f \triangleq g-\ell$ expressed as the difference between a monotone submodular function $g$ and a modular function $\ell$. Submodularity is inherently related to the notions of diversity, coverage, and representativeness. In particular, finding the mode (i.e., the most likely configuration) of many popular probabilistic models of diversity, such as determinantal point processes and strongly log-concave distributions, involves maximization of (regularized) submodular functions. Since a regularized function $f$ can potentially take on negative values, the classic theory of submodular maximization, which heavily relies on the non-negativity assumption of submodular functions, is not applicable. To circumvent this challenge, we develop the first one-pass streaming algorithm for maximizing a regularized submodular function subject to a $k$-cardinality constraint. Furthermore, we develop the first distributed algorithm that returns a solution $S$ in $O(1/ \epsilon)$ rounds of MapReduce computation. We highlight that our result, even for the unregularized case where the modular term $\ell$ is zero, improves the memory and communication complexity of the state-of-the-art by a factor of $O(1/ \epsilon)$ while arguably provides a simpler distributed algorithm and a unifying analysis. We empirically study the performance of our scalable methods on a set of real-life applications, including finding the mode of negatively correlated distributions, vertex cover of social networks, and several data summarization tasks.

Vehicle detection and tracking play an important role in autonomous vehicles and intelligent transportation systems. Adverse weather conditions such as the presence of heavy snow, fog, rain, dust or sandstorm situations are dangerous restrictions on camera’s function by reducing visibility, affecting driving safety. Indeed, these restrictions impact the performance of detection and tracking algorithms utilized in the traffic surveillance systems and autonomous driving applications. In this article, we start by proposing a visibility enhancement scheme consisting of three stages: illumination enhancement, reflection component enhancement, and linear weighted fusion to improve the performance. Then, we introduce a robust vehicle detection and tracking approach using a multi-scale deep convolution neural network. The conventional Gaussian mixture probability hypothesis density filter based tracker is utilized jointly with hierarchical data associations (HDA), which splits into detection-to-track and track-to-track associations. Herein, the cost matrix of each phase is solved using the Hungarian algorithm to compensate for the lost tracks caused by missed detection. Only detection information (i.e., bounding boxes with detection scores) is used in HDA without visual features information for rapid execution. We have also introduced a novel benchmarking dataset designed for research in applications of autonomous vehicles under adverse weather conditions called DAWN. It consists of real-world images collected with different types of adverse weather conditions. The proposed method is tested on DAWN, KITTI, and MS-COCO datasets and compared with 21 vehicle detectors. Experimental results have validated effectiveness of the proposed method which outperforms state-of-the-art vehicle detection and tracking approaches under adverse weather conditions.

Signal decomposition is a classical problem in signal processing, which aims to separate an observed signal into two or more components, each with its own property. Usually, each component is described by its own subspace or dictionary. Extensive research has been done for the case where the components are additive, but in real-world applications, the components are often non-additive. For example, an image may consist of a foreground object overlaid on a background, where each pixel either belongs to the foreground or the background. In such a situation, to separate signal components, we need to find a binary mask which shows the location of each component. Therefore, it requires solving a binary optimization problem. Since most of the binary optimization problems are intractable, we relax this problem to the approximated continuous problem and solve it by alternating optimization technique. We show the application of the proposed algorithm for three applications: separation of text from a background in images, separation of moving objects from a background undergoing global camera motion in videos, and separation of sinusoidal and spike components in 1-D signals. We demonstrate in each case that considering the non-additive nature of the problem can lead to a significant improvement.

In this paper, we propose bag of adversarial features (BAFs) for identifying mild traumatic brain injury (MTBI) patients from their diffusion magnetic resonance images (MRIs) (obtained within one month of injury) by incorporating unsupervised feature learning techniques. MTBI is a growing public health problem with an estimated incidence of over 1.7 million people annually in USA. Diagnosis is based on clinical history and symptoms, and accurate, concrete measures of injury are lacking. Unlike most of the previous works, which use hand-crafted features extracted from different parts of brain for MTBI classification, we employ feature learning algorithms to learn more discriminative representation for this task. A major challenge in this field thus far is the relatively small number of subjects available for training. This makes it difficult to use an end-to-end convolutional neural network to directly classify a subject from MRIs. To overcome this challenge, we first apply an adversarial auto-encoder (with convolutional structure) to learn patch-level features, from overlapping image patches extracted from different brain regions. We then aggregate these features through a bag-of-words approach. We perform an extensive experimental study on a dataset of 227 subjects (including 109 MTBI patients, and 118 age and sex-matched healthy controls) and compare the bag-of-deep-features with several previous approaches. Our experimental results show that the BAF significantly outperforms earlier works relying on the mean values of MR metrics in selected brain regions.

Single Image Super Resolution (SISR) is a well-researched problem with broad commercial relevance. However, most of the SISR literature focuses on small-size images under 500px, whereas business needs can mandate the generation of very high resolution images. At Expedia Group, we were tasked with generating images of at least 2000px for display on the website - four times greater than the sizes typically reported in the literature. This requirement poses a challenge that state-of-the-art models, validated on small images, have not been proven to handle. In this paper, we investigate solutions to the problem of generating high-quality images for large-scale1super resolution in a commercial setting. We find that training a generative adversarial network (GAN) with attention from scratch using a large-scale lodging image data set generates images with high PSNR and SSIM scores. We describe a novel attentional SISR model for large-scale images, A-SRGAN, that uses a Flexible Self Attention layer to enable processing of large-scale images. We also describe a distributed algorithm which speeds up training by around a factor of five.

Palmprint recognition has drawn a lot of attentions during recent years. Different features and algorithms have been proposed for palmprint recognition in the past such as Gabor-based features, wavelet features, and histogram of oriented lines. In this paper, a powerful image representation, so called deep scattering network, is used for recognition. Scattering network is a convolutional network where its architecture and filters are predefined wavelet transforms. Scattering transform is designed such that the features in its first layer are similar to SIFT descriptors and the higher layers' features capture higher frequency content of the signal which are lost in SIFT. After extraction of scattering features, their dimensionality is reduced by applying principal component analysis (PCA). By doing so, a great amount of computation complexity can be reduced. At the end, the recognition is performed using two different classifiers, multi-class SVM and minimum-distance classifier. The proposed scheme has been tested on a well-known palmprint database and achieved accuracy rates of 99.4% and 99.9% using minimum distance classifier and SVM respectively, outperforming previous algorithms on this dataset.

Subspace learning is an important problem, which has many applications in image and video processing. It can be used to find a low-dimensional representation of signals and images. But in many applications, the desired signal is heavily distorted by outliers and noise, which negatively affect the learned subspace. In this work, we present a novel algorithm for learning a subspace for signal representation, in the presence of structured outliers and noise. The proposed algorithm tries to jointly detect the outliers and learn the subspace for images. We present an alternating optimization algorithm for solving this problem, which iterates between learning the subspace and finding the outliers. This algorithm has been trained on a large number of image patches, and the learned subspace is used for image segmentation, and is shown to achieve better segmentation results than prior methods, including least absolute deviation fitting, k-means clustering based segmentation in DjVu, and shape primitive extraction and coding algorithm.

Sparse decomposition has been widely used for different applications, such as source separation, image classification, image denoising and more. This paper presents a new algorithm for segmentation of an image into background and foreground text and graphics using sparse decomposition and total variation minimization. The proposed method is designed based on the assumption that the background part of the image is smoothly varying and can be represented by a linear combination of a few smoothly varying basis functions, while the foreground text and graphics can be modeled with a sparse component overlaid on the smooth background. The background and foreground are separated using a sparse decomposition framework regularized with a few suitable regularization terms which promotes the sparsity and connectivity of foreground pixels. This algorithm has been tested on a dataset of images extracted from HEVC standard test sequences for screen content coding, and is shown to have superior performance over some prior methods, including least absolute deviation fitting, k-means clustering based segmentation in DjVu and shape primitive extraction and coding (SPEC) algorithm.

We propose an algorithm for separating the foreground (mainly text and line graphics) from the smoothly varying background in screen content images. The proposed method is designed based on the assumption that the background part of the image is smoothly varying and can be represented by a linear combination of a few smoothly varying basis functions, while the foreground text and graphics create sharp discontinuity and cannot be modeled by this smooth representation. The algorithm separates the background and foreground using a least absolute deviation method to fit the smooth model to the image pixels. This algorithm has been tested on several images from HEVC standard test sequences for screen content coding, and is shown to have superior performance over other popular methods, such as k-means clustering based segmentation in DjVu and shape primitive extraction and coding (SPEC) algorithm. Such background/foreground segmentation are important pre-processing steps for text extraction and separate coding of background and foreground for compression of screen content images.