Using multi-modal data to learn region representations has gained popularity for its ability to reveal diverse socioeconomic features in cities. However, many studies focus solely on semantic features from points-of-interest (POIs), neglecting the issue of spatial imbalance. This article introduces a Multi-Graph Representation Learning framework for Region Embedding (MGRL4RE), which leverages both inter-region and intra-region correlations through two main components: multi-graph construction based on various region correlations and multi-graph representation learning. The construction module creates a multi-graph reflecting various correlations among regions, utilizing geo-tagged POIs, region data, and human mobility data. Specifically, we assess a region’s importance relative to its spatial context (neighborhood) and develop spatially invariant semantic features to address spatial imbalance. Furthermore, the representation learning module generates comprehensive and effective region representations via multi-view embedding fusion. Our extensive experiments across various downstream tasks, including land use clustering, region popularity prediction, and crime prediction, confirm that our model significantly outperforms existing state-of-the-art region embedding methods.

With the expansion and growth of cities, profiling urban areas with the advent of multi-modal urban datasets (e.g., points-of-interest and street view imagery) has become increasingly important in urban planing and management. Particularly, street view images have gained popularity for understanding the characteristics of urban areas due to its abundant visual information and inherent correlations with human activities. In this study, we define a street segment represented by multiple street view images as the minimum spatial unit for analysis and predict its functional and socioeconomic indicators, which presents several challenges in modeling spatial distributions of images on a street and the spatial topology (adjacency) of streets. Meanwhile, Large Language Models are capable of understanding imagery data based on its extraordinary knowledge base and unveil a remarkable opportunity for profiling streets with images. In view of the challenges and opportunity, we present a semi-supervised Urban Street Profiling Model (USPM) based on street view imagery and spatial adjacency of urban streets. Specifically, given a street with multiple images, we first employ a newly designed spatial context-based contrastive learning method to generate feature vectors of images and then apply the LSTM-based fusion method to encode multiple images on a street to yield the street visual representation; we then create the descriptions of street scenes for street view images based on the SPHINX (a large language model) and produce the street textual representation; finally, we build an urban street graph based on spatial topology (adjacency) and employ a semi-supervised graph learning algorithm to further encode the street representations for prediction. We conduct thorough experiments with real-world datasets to assess the proposed USPM. The experimental results demonstrate that USPM considerably outperforms baseline methods in two urban prediction tasks.

Weather forecasting is of great importance for human life and various real-world fields, e.g., traffic prediction, agricultural production, and tourist industry. Existing methods can be roughly divided into two categories: theory-driven (e.g., numerical weather prediction (NWP)) and data-driven methods. Theory-driven methods require a complex simulation of the physical evolution process in the atmosphere model using supercomputers, while most data-driven methods learn the underlying laws from the historical weather records via deep learning models. However, some data-driven methods simply regard all weather variables of monitoring stations as a whole and fail to more granularly exploit complex correlations across different stations, while others prefer to construct large neural networks with massive learnable parameters. To alleviate these defects, we propose a spatio-temporal contrastive self-supervision method and a generative contextual self-supervised technique to capture spatial and temporal dependencies from the station-level and variable-level, respectively. Through these well-designed self-supervised tasks, uncomplicated networks obtain strong capability to capture latent representations for weather changes with time-varying. Thereafter, an effective encoder-decoder based fine-tuning framework is proposed, consisting of three self-supervised encoders. Extensive experiments conducted on four real-world weather condition datasets demonstrate that our method outperforms the state-of-the-art models and also empirically validates the feasibility of each self-supervised task.

Deep semi-supervised learning (SSL) methods aim to utilize abundant unlabeled data to improve the seen-class classification. However, in the open-world scenario, collected unlabeled data tend to contain unseen-class data, which would degrade the generalization to seen-class classification. Formally, we define the problem as safe deep semi-supervised learning with unseen-class unlabeled data. One intuitive solution is removing these unseen-class instances after detecting them during the SSL process. Nevertheless, the performance of unseen-class identification is limited by the lack of suitable score function, the uncalibrated model, and the small number of labeled data. To this end, we propose a safe SSL method called SAFER-STUDENT from the teacher-student view. First, to enhance the ability of teacher model to identify seen and unseen classes, we propose a general scoring framework calledDiscrepancy withRaw (DR). Second, based on unseen-class data mined by teacher model from unlabeled data, we calibrate student model by newly proposedUnseen-classEnergy-boundedCalibration (UEC) loss. Third, based on seen-class data mined by teacher model from unlabeled data, we proposeWeightedConfirmationBiasElimination (WCBE) loss to boost seen-class classification of student model. Extensive studies show that SAFER-STUDENT remarkably outperforms the state-of-the-art, verifying the effectiveness of our method in the under-explored problem.

Fine-grained urban flow inference focuses on inferring fine-grained urban flows based solely on coarse-grained observations, which is essential for the city management and transportation services. However, most of the existing methods assume that partial urban flows in coarse-grained regions cannot be observable. In this study, we propose a multi-task framework known as UrbanSTA with space-time attraction learning to estimate missing values in coarse-grained urban flow map and forecast fine-grained urban flows simultaneously. Specifically, UrbanSTA comprises two parts: the flow completion network STA and the fine-grained flow inference network FIN. STA captures space-time features with a separable space-time attention encoder and recovers the missing flow features with a decoder. FIN directly uses complete coarse-grained flow features for further decoding, and reconstructs fine-grained flow features based on the complex associations between coarse- and fine-grained urban flows, relying on upsampling constraints. Extensive experiments conducted on two real-world datasets demonstrate that our proposed model yields the best results compared to other state-of-the-art methods. The source code has been provided at https://github.com/Wangzheaos/UrbanSTA.

As a developing trend of urbanization, massive amounts of urban statistical data with multiple views (e.g., views of Population and Economy) are increasingly collected and benefited to diverse domains, including transportation service, regional analysis, etc. Unfortunately, these statistical data that are divided into fine-grained regions usually suffer from missing value problem during the acquisition and storage processes. It is mianly caused by some inevitable circumstances, e.g., the document defacement, statistical difficulty in remote districts, and inaccurate information cleaning, etc. Those missing entries which make valuable information invisible may distort the further urban analysis. To improve the quality of missing data imputation, we propose an improved spatial multi-kernel learning method to guide the imputation process incorporating with the adaptive-weight non-negative matrix factorization strategy. Our model takes into account the regional latent similarities and the real geographical positions as well as the correlations among various views that are able to complete missing values precisely. We conduct intensive experiments to evaluate our method and compare with other state-of-the-art approaches on real-world datasets. All the empirical results show that the proposed model outperforms all the other state-of-the-art methods. Additionally, our model represents a strong generalization ability across multiple cities.

As a critical task of the urban traffic services, fine-grained urban flow inference (FUFI) benefits in many fields including intelligent transportation management, urban planning, public safety. FUFI is a technique that focuses on inferring fine-grained urban flows depending solely on observed coarse-grained data. However, existing methods always require massive learnable parameters and the complex network structures. To reduce these defects, we formulate a contrastive self-supervision method to predict fine-grained urban flows taking into account all correlated spatial and temporal contrastive patterns. Through several well-designed self-supervised tasks, uncomplicated networks have a strong ability to capture high-level representations from flow data. Then, a fine-tuning network combining with three pre-training encoder networks is proposed. We conduct experiments to evaluate our model and compare with other state-of-the-art methods by using two real-world datasets. All the empirical results not only show the superiority of our model against other comparative models, but also demonstrate its effectiveness in the resource-limited environment.

Zero-shot hashing targets to learn the hash codes of images in unseen classes based on the limited training data provided by seen classes. In zero-shot hashing, transferring the supervised knowledge, such as attributes and semantic relations, from seen classes to unseen ones is a widely employed method, where the performance is always subject to the ability to capture these supervised knowledge (which is always difficult to obtain). Therefore, in this study, we propose a new methodology for zero-shot hashing via an asymmetric ratio similarity matrix (ASZH), which only needs to calculate the semantic similarity among seen classes for hash learning. Specifically, we use an asymmetric ratio matrix in the similarity calculation to further explore the influence of similarity, where the values of positive weights for similar samples are not equivalent to those of negative ones for dissimilar samples. Additionally, a theoretical analysis regarding the utilization of an asymmetric ratio matrix is provided in this study. The experiments on three large benchmark datasets indicate that the proposed method achieves excellent performance than several state-of-the-art hashing methods.

Venue categories used in location-based social networks often exhibit a hierarchical structure, together with the category sequences derived from users’ check-ins. The two data modalities provide a wealth of information for us to capture the semantic relationships between those categories. To understand the venue semantics, existing methods usually embed venue categories into low-dimensional spaces by modeling the linear context (i.e., the positional neighbors of the given category) in check-in sequences. However, the hierarchical structure of venue categories, which inherently encodes the relationships between categories, is largely untapped. In this article, we propose a venue C ategory E mbedding M odel named Hier-CEM , which generates a latent representation for each venue category by embedding the Hier archical structure of categories and utilizing multiple types of context. Specifically, we investigate two kinds of hierarchical context based on any given venue category hierarchy and show how to model them together with the linear context collaboratively. We apply Hier-CEM to three tasks on two real check-in datasets collected from Foursquare. Experimental results show that Hier-CEM is better at capturing both semantic and sequential information inherent in venues than state-of-the-art embedding methods.

Recent studies have verified that learning compact hash codes can facilitate big data retrieval processing. In particular, learning the deep hash function can greatly improve the retrieval performance. However, the existing deep supervised hashing algorithm treats all the samples in the same way, which leads to insufficient learning of difficult samples. Therefore, we cannot obtain the accurate learning of the similarity relation, making it difficult to achieve satisfactory performance. In light of this, this work proposes a deep supervised hashing model, called deep adaptive attention triple hashing (DAATH), which weights the similarity prediction scores of positive and negative samples in the form of triples, thus giving different degrees of attention to different samples. Compared with the traditional triple loss, it places a greater emphasis on the difficult triple, dramatically reducing the redundant calculation. Extensive experiments have been conducted to show that DAAH consistently outperforms the state-of-the-arts, confirmed its the effectiveness.

Convolutional Neural Network (CNN) based multi-task learning methods have been widely used in a variety of applications of computer vision. Towards effective multi-task CNN architectures, recent studies automatically learn the optimal combinations of task-specific features at single network layers. However, they generally construct an unchanged operation of feature aggregation after training, regardless of the characteristics of input features. In this paper, we propose a novel Adaptive Feature Aggregation (AFA) layer for multi-task CNNs, in which a dynamic aggregation mechanism is designed to allow each task to adaptively determine the degree to which the feature aggregation of different tasks is needed according to the feature dependencies. On both pixel-level and image-level tasks, we demonstrate that our approach significantly outperforms the previous state-of-the-art methods of multi-task CNNs.

Student performance prediction aims to leverage student-related information to predict their future academic outcomes, which may be beneficial to numerous educational applications, such as personalized teaching and academic early warning. In this paper, we seek to address the problem by analyzing students' daily studying and living behavior, which is comprehensively recorded via campus smart cards. Different from previous studies, we propose an end-to-end student performance prediction model, namely Tri-branch CNN, which is equipped with three types of convolutional filters, i.e., the row-wise convolution, column-wise convolution, and group-wise convolution, to effectively capture the duration, periodicity, and location-aware characteristic of student behavior, respectively. We also introduce the attention mechanism and cost-sensitive learning strategy to further improve the accuracy of our approach. Extensive experiments on a large-scale real-world dataset demonstrate the potential of our approach for student performance prediction.

Multi-view hashing can well support large-scale near-duplicate video retrieval, due to its desirable advantages of mutual reinforcement of multiple features, low storage cost, and fast retrieval speed. However, there are still two limitations that impede its performance. First, existing methods only consider local structures in multiple features. They ignore the global structure that is important for near-duplicate video retrieval, and cannot fully exploit the dependence and complementarity of multiple features. Second, existing works always learn hashing functions bit by bit, which unfortunately increases the time complexity of hash function learning. In this paper, we propose a supervised hashing scheme, termed as joint multi-view hashing (JMVH), to address the aforementioned problems. It jointly preserves the global and local structures of multiple features while learning hashing functions efficiently. Specially, JMVH considers features of video as items, based on which an underlying Hamming space is learned by simultaneously preserving their local and global structures. In addition, a simple but efficient multi-bit hash function learning based on generalized eigenvalue decomposition is devised to learn multiple hash functions within a single step. It can significantly reduce the time complexity of conventional hash function learning processes that sequentially learn multiple hash functions bit by bit. The proposed JMVH is evaluated on two public databases: CC_WEB_VIDEO and UQ_VIDEO. Experimental results demonstrate that the proposed JMVH achieves more than a 5 percent improvement compared to several state-of-the-art methods which indicates the superior performance of JMVH.

The linear model is commonly utilized in hashing methods owing to its efficiency. To obtain better accuracy, linear-based hashing methods focus on designing a generalized linear objective function with different constraints or penalty terms that consider neighborhood information. In this study, we propose a novel generalized framework called Model Boost (MoBoost), which can achieve the self-improvement of the linear-based hashing. The proposed MoBoost is used to improve model parameter optimization for linear-based hashing methods without adding new constraints or penalty terms. In the proposed MoBoost, given a linear-based hashing method, we first execute the method several times to get several different hash codes for training samples, and then combine these different hash codes into one set utilizing one novel fusion strategy. Based on this set of hash codes, we learn some new parameters for the linear hash function that can significantly improve accuracy. The proposed MoBoost can be generally adopted in existing linear-based hashing methods, achieving more precise and stable performance compared to the original methods while imposing negligible added expenditure in terms of time and space. Extensive experiments are performed based on three benchmark datasets, and the results demonstrate the superior performance of the proposed framework.

Automatically assessing image quality from an aesthetic perspective is of great interest to the high-level vision research community. Existing methods are typically non-personalized and quantify image aesthetics with a universal label. However, given the fact that aesthetics is a subjective perception, how to understand user aesthetic perceptions poses a formidable challenge to image aesthetics assessment. In this paper, we propose to model user aesthetic perceptions using a set of exemplar images from social media platforms, and realize personalized aesthetics assessment by transferring this knowledge to adapt the results of the trained generic model. In this way, image aesthetics is measured from both aspects of visual quality and user tastes. Extensive experiments on two benchmark datasets well verified the potential of our approach for personalized image aesthetics assessment.

Finger vein image retrieval is one significant technique for performing fast identification especially in large-scale applications. However, most existing retrieval methods were based on fixed-scale feature of non-overlapped rectangular image block, in which the representation ability of feature and the local consistency of vein pattern were both overlooked. And the weak encoding (e.g., predefined threshold based binarization) was also limited the retrieval performance. Focusing on these problems, this paper proposes a novel finger vein image retrieval framework based on similarity-preserving encoding of scale-varied superpixel feature. In the framework, locally consistent pixels in one superpixel are used as a unit of feature representation, and the feature length is varied with the category of the superpixel classified by the variance of lowest dimensional feature. Additionally, the feature compaction and feature rotation based encoding can minimize the quantization loss and preserve the similarity between the scale-varied feature and the encoded binary codes. Experimental results on six public finger vein databases demonstrate that the superiority of the proposed coding scale-varied superpixel feature based retrieval approach over the state-of-the-arts.

Aesthetics has become increasingly prominent for image search to enhance user satisfaction. Therefore, image aesthetics assessment is emerging as a promising research topic in recent years. In this paper, distinguished from existing studies relying on a single label, we propose to quantify the image aesthetics by a distribution over quality levels. The distribution representation can effectively characterize the disagreement among the aesthetic perceptions of users regarding the same image. Our framework is developed on the foundation of label distribution learning, in which the reliability of training examples and the correlations between quality levels are fully taken into account. Extensive experiments on two benchmark datasets well verified the potential of our approach for aesthetics assessment. The role of aesthetics in image search was also rigorously investigated.

In this paper, we introduce polygene-based evolution, a novel framework for evolutionary algorithms (EAs) that features distinctive operations in the evolution process. In traditional EAs, the primitive evolution unit is gene, where genes are independent components during evolution. In polygene-based evolutionary algorithms (PGEAs), the evolution unit is polygene, i.e., a set of co-regulated genes. Discovering and maintaining quality polygenes can play an effective role in evolving quality individuals. Polygenes generalize genes, and PGEAs generalize EAs. Implementing the PGEA framework involves three phases: polygene discovery, polygene planting, and polygene-compatible evolution. Extensive experiments on function optimization benchmarks in comparison with the conventional and state-of-the-art EAs demonstrate the potential of the approach in accuracy and efficiency improvement.

Importance weighted active learning (IWAL) introduces a weighting scheme to measure the importance of each instance for correcting the sampling bias of the probability distributions between training and test datasets. However, the weighting scheme of IWAL involves the distribution of the test data, which can be straightforwardly estimated in active learning by interactively querying users for labels of selected test instances, but difficult for conventional learning where there are no interactions with users, referred as passive learning. In this paper, we investigate the insufficient sampling bias problem, i.e., bias occurs only because of insufficient samples, but the sampling process is unbiased. In doing this, we present two assumptions on the sampling bias, based on which we propose a practical weighting scheme for the empirical loss function in conventional passive learning, and present IWPL, an importance weighted passive learning framework. Furthermore, we provide IWSVM, an importance weighted SVM for validation. Extensive experiments demonstrate significant advantages of IWSVM on benchmarks and synthetic datasets.