With the rapid progress of urbanization, city governance faces growing challenges such as traffic violations and environmental pollution. Traditional manual monitoring methods are inefficient and costly. To enhance the efficiency of monitoring and managing uncivil behaviors in urban environments, we propose a self-supervised video hashing retrieval framework for uncivil behavior recognition. Leveraging deep learning techniques, our method generates compact binary hash codes for both video and text modalities via a contrastive masked autoencoder (ConMH), enabling efficient large-scale retrieval. We further improve ConMH by introducing cross-attention mechanisms in the text hashing branch to better handle context dependencies. To optimize retrieval results, we integrate five multimodal fusion and ranking strategies, including a novel Hybrid Distance-Rank Fusion method that balances similarity scores and rank information. Experiments conducted on MSRVTT and MSVD datasets demonstrate that our approach achieves superior performance in mAP@K and NDCG metrics. The framework significantly enhances cross-modal semantic coverage, ensures high retrieval precision, and maintains low computational and storage overhead through binary encoding.

Memory-resident malware has become a huge threat to cybersecurity. They perform malicious operations only in memory and are difficult to detect by existing technologies. Existing malware detection solutions fail to effectively extract API sequence’s semantic features and memory data features related to malicious behaviors in memory dumps. This research paper presents a novel detection framework to address these limitations. It first extracts intrinsic semantic features of API sequences from memory data using a fine-tuned BERT, then extracts executable data features from memory dumps using a pre-trained ResNet34 neural network. It then splices the two features to train a deep neural network-based detection model. We created a high-quality dataset with 2180 benign programs and 1897 recent memory-resident malware samples. We implement MemAPIDet for Windows 10. It performs better than the state-of-the-art methods with a prediction accuracy of 97.78%

Zequan Liu, Jiawen Lyn, Wei Zhu, Xing Tian, Yvette Graham. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2024.

Abstract As the proliferation of video content continues, and many video archives lack suitable metadata, therefore, video retrieval, particularly through example‐based search, has become increasingly crucial. Existing metadata often fails to meet the needs of specific types of searches, especially when videos contain elements from different modalities, such as visual and audio. Consequently, developing video retrieval methods that can handle multi‐modal content is essential. An innovative Multi‐modal Video Search by Examples (MVSE) framework is introduced, employing state‐of‐the‐art techniques in its various components. In designing MVSE, the authors focused on accuracy, efficiency, interactivity, and extensibility, with key components including advanced data processing and a user‐friendly interface aimed at enhancing search effectiveness and user experience. Furthermore, the framework was comprehensively evaluated, assessing individual components, data quality issues, and overall retrieval performance using high‐quality and low‐quality BBC archive videos. The evaluation reveals that: (1) multi‐modal search yields better results than single‐modal search; (2) the quality of video, both visual and audio, has an impact on the query precision. Compared with image query results, audio quality has a greater impact on the query precision (3) a two‐stage search process (i.e. searching by Hamming distance based on hashing, followed by searching by Cosine similarity based on embedding); is effective but increases time overhead; (4) large‐scale video retrieval is not only feasible but also expected to emerge shortly.

Abstract The surface quality of aluminium alloy castings is crucial to quality control. Aiming to address the challenges of limited samples and extensive computation in deep learning‐based surface defect detection for aluminium alloy castings, this paper proposes a surface defect detection method based on data enhancement and the Casting Real‐Time DEtection TRansformer. First, to tackle the issue of small sample sizes and uneven distribution in surface defect data sets of aluminium alloy castings, ECA‐MetaAconC Deep Convolution Generative Adversarial Networks is proposed for generating defects with fewer samples and employ the image augmentation (IMGAUG) library for sample enhancement. Second, building upon the Real‐Time DEtection TRansformer (RT‐DETR), a lightweight partial‐rep convolution is designed to decrease the network's parameter count. Simultaneously, the Deformable attention module and the DRBC3 module are introduced to enhance the neck network, thereby improving the model's capability to capture information and enhancing its detection performance. Compared to RT‐DETR, this method reduces the number of model parameters by 38.7%, increases mAP by 1.5%, and achieves a frame rate that is 1.58 times higher than the original model. The experimental results demonstrate that this method can effectively and accurately detect surface defects in aluminium alloy castings, satisfying industrial requirements.

Binary hashing is an effective approach for content-based image retrieval, and learning binary codes with neural networks has attracted increasing attention in recent years. However, the training of hashing neural networks is difficult due to the binary constraint on hash codes. In addition, neural networks are easily affected by input data with small perturbations. Therefore, a sensitive binary hashing autoencoder (SBHA) is proposed to handle these challenges by introducing stochastic sensitivity for image retrieval. SBHA extracts meaningful features from original inputs and maps them onto a binary space to obtain binary hash codes directly. Different from ordinary autoencoders, SBHA is trained by minimizing the reconstruction error, the stochastic sensitive error, and the binary constraint error simultaneously. SBHA reduces output sensitivity to unseen samples with small perturbations from training samples by minimizing the stochastic sensitive error, which helps to learn more robust features. Moreover, SBHA is trained with a binary constraint and outputs binary codes directly. To tackle the difficulty of optimization with the binary constraint, we train the SBHA with alternating optimization. Experimental results on three benchmark datasets show that SBHA is competitive and significantly outperforms state-of-the-art methods for binary hashing.

Self-supervised video hashing methods retrieve large-scale video data without labels by making full use of visual and temporal information in original videos. Existing methods are not robust enough to handle small temporal differences between similar videos, because of the ignoring of future unseen samples on temporal which leads to large generalization errors. At the same time, existing self-supervised methods cannot preserve pairwise similarity information between large-scale unlabeled data efficiently and effectively. Thus, a self-supervised temporal sensitive video hashing (TSVH) is proposed in the paper for video retrieval. The TSVH uses a transformer-based autoencoder network with temporal sensitivity regularization to achieve low sensitivity of local temporal perturbations and preserve information of global temporal sequence. The pairwise similarity between video samples is effectively preserved by applying a hashing-based affinity matrix in the method. Experiments on realistic datasets show that the TSVH outperforms several state-of-the-art methods and classic methods.

Hashing methods are widely used for content-based image retrieval due to their attractive time and space efficiencies. Several dynamic hashing methods have been proposed for image retrieval tasks in non-stationary environments. However, concept drift problems in non-stationary environment are seldomly considered which lead to significant deterioration of performance. Therefore, we propose Deep Incremental Hashing (DIH). For the learning part, similarity-preserving object codes of each newly arriving data chunk are computed using the product of its label matrix and a random Gaussian matrix generated offline. A point-wise loss function is then devised to guide the learning of a deep hash neural network. To retain the learned knowledge of former chunks, a weighting-based method is utilized to combine different hash tables trained at different time steps to form a multi-table hashing system. Experimental results on 13 simulated concept drift environments show that DIH adapts to non-stationary data environments well and yields better retrieval performance than existing dynamic hashing methods.

Deep hashing has proven to be efficient and effective for large-scale face retrieval. However, existing hashing methods are designed for normal face images only. They fail to consider the fact that face images may be occluded because of wearing masks, hats, glasses, etc. Retrieval performance of existing face retrieval methods is much worse when dealing with occluded face images. In this work, we propose the knowledge distillation hashing (KDH) to deal with occluded face images. The KDH is a two-stage learning approach with teacher-student model distillation. We first train a teacher hashing network using normal face images and then the knowledge from teacher model is used to guide the optimization of the student model using occluded face images as input only. With knowledge distillation, we build a connection between imperfect face information and the optimal hash codes. Experimental results show that the KDH yields significant improvements and better retrieval performance in comparison to existing state-of-the-art deep hashing retrieval methods under six different face occlusion situations.

In practical active noise control systems, saturation nonlinearity is one of the most common nonlinear effects, which can occur at the electronic circuits in the secondary path. Adaptive algorithms designed for linear systems, such as the filtered-x least mean square (FxLMS) algorithm, may diverge in nonlinear systems with a saturated secondary path and result in limited noise reduction performance. Various nonlinear algorithms with improved stability have been proposed; however, a trade-off between noise reduction performance and computational complexity exists when selecting a satisfactory algorithm. This paper therefore proposes a simple algorithm that can achieve significant noise reduction with low computational complexity. The proposed algorithm adopts the same update equation as the FxLMS algorithm, but uses an intermittent update strategy to avoid unnecessary fluctuations of the weight vector when the output of the control filter lies in the saturation region. Simulation results under different conditions demonstrate the effectiveness and advantages of the proposed algorithm.

Undersampling is a popular method to solve imbalanced classification problems. However, sometimes it may remove too many majority samples which may lead to loss of informative samples. In this article, the hashing-based undersampling ensemble (HUE) is proposed to deal with this problem by constructing diversified training subspaces for undersampling. Samples in the majority class are divided into many subspaces by a hashing method. Each subspace corresponds to a training subset which consists of most of the samples from this subspace and a few samples from surrounding subspaces. These training subsets are used to train an ensemble of classification and regression tree classifiers with all minority class samples. The proposed method is tested on 25 UCI datasets against state-of-the-art methods. Experimental results show that the HUE outperforms other methods and yields good results on highly imbalanced datasets.

Current hashing-based image retrieval methods mostly assume that the database of images is static. However, this assumption is not true in cases where the databases are constantly updated (e.g., on the Internet) and there exists the problem of concept drift. The online (also known as incremental) hashing methods have been proposed recently for image retrieval where the database is not static. However, they have not considered the concept drift problem. Moreover, they update hash functions dynamically by generating new hash codes for all accumulated data over time which is clearly uneconomical. In order to solve these two problems, concept preserving hashing (CPH) is proposed. In contrast to the existing methods, CPH preserves the original concept, that is, the set of hash codes representing a concept is preserved over time, by learning a new set of hash functions to yield the same set of hash codes for images (old and new) of a concept. The objective function of CPH learning consists of three components: 1) isomorphic similarity; 2) hash codes partition balancing; and 3) heterogeneous similarity fitness. The experimental results on 11 concept drift scenarios show that CPH yields better retrieval precisions than the existing methods and does not need to update hash codes of previously stored images.

Concept drift is prevalent in non-stationary data environments but is rarely researched in image retrieval. Therefore, more research is needed on image retrieval in non-stationary data environments so that highly relevant images can still be retrieved when concept drifts happen. Hashing is a key technique to allow efficient image retrieval, so incremental hashing technique emerges in recent years for image retrieval in non-stationary environments. A state-of-the-art method isIncremental Hashing(ICH). ICH trains new hash tables on new data without considering the performance of previous hash tables, so the dependency of successive hash tables is ignored. To make use of this dependency in order to improve the performance of image retrieval in non-stationary environments,Complementary Incremental Hashing with query-adaptive Re-ranking(CIHR) is proposed in this paper. CIHR trains multiple hash tables incrementally, one for each data chunk of images. A new hash table is trained on a new data chunk of images as well as those images badly hashed by previous hash tables, thus the new hash table is complementary to the previous hash tables. To use the hash tables more effectively, a query-adaptive re-ranking method is used to weight all hash functions in each hash table according to their retrieval performance with respect to a given query. Weighted Hamming distance is finally used to evaluate the similarity between the query and the images in the database, as the basis of image retrieval. Experimental results on simulated non-stationary scenarios show that the proposed CIHR method achieves higher retrieval accuracy than all methods being compared, thus setting a new state of the art in image retrieval in non-stationary data environments.

The probabilistic classification vector machine (PCVM) synthesizes the advantages of both the support vector machine and the relevant vector machine, delivering a sparse Bayesian solution to classification problems. However, the PCVM is currently only applicable to binary cases. Extending the PCVM to multiclass cases via heuristic voting strategies such as one-vs-rest or one-vs-one often results in a dilemma where classifiers make contradictory predictions, and those strategies might lose the benefits of probabilistic outputs. To overcome this problem, we extend the PCVM and propose a multiclass PCVM (mPCVM). Two learning algorithms, i.e., one top-down algorithm and one bottom-up algorithm, have been implemented in the mPCVM. The top-down algorithm obtains the maximum a posteriori (MAP) point estimates of the parameters based on an expectation-maximization algorithm, and the bottom-up algorithm is an incremental paradigm by maximizing the marginal likelihood. The superior performance of the mPCVMs, especially when the investigated problem has a large number of classes, is extensively evaluated on the synthetic and benchmark data sets.

Most of current hashing methods are proposed based on the assumption that the database is stationary. However, this assumption is not always true as the data environment is sometimes non-stationary. When new images being added to the database, data distributions of existing classes may change and new classes may also appear which result in concept drifts. The problem of concept drifts is unavoidable in non-stationary data environments. Incremental Hashing (ICH) is an effective method for image retrieval in non-stationary data environments with concept drifts using multiple hash tables. In ICH, new concept is adapted by training new hash table using the most updated data chunks. However, images in the new data chunk may not be all informative for updating. To enhance the efficiency of ICH, ICH with Undersampling (ICHUS) is proposed to select informative samples in the new data chunk for the training of new hash table to adapt to the non-stationary data environment. Experimental results show that ICHUS yields a better retrieval performance than ICH and state-of-art non-stationary hashing methods.

Images are uploaded to the Internet over time which makes concept drifting and distribution change in semantic classes unavoidable. Current hashing methods being trained using a given static database may not be suitable for nonstationary semantic image retrieval problems. Moreover, directly retraining a whole hash table to update knowledge coming from new arriving image data may not be efficient. Therefore, this paper proposes a new incremental hash-bit learning method. At the arrival of new data, hash bits are selected from both existing and newly trained hash bits by an iterative maximization of a 3-component objective function. This objective function is also used to weight selected hash bits to re-rank retrieved images for better semantic image retrieval results. The three components evaluate a hash bit in three different angles: 1) information preservation; 2) partition balancing; and 3) bit angular difference. The proposed method combines knowledge retained from previously trained hash bits and new semantic knowledge learned from the new data by training new hash bits. In comparison to table-based incremental hashing, the proposed method automatically adjusts the number of bits from old data and new data according to the concept drifting in the given data via the maximization of the objective function. Experimental results show that the proposed method outperforms existing stationary hashing methods, table-based incremental hashing, and online hashing methods in 15 different simulated nonstationary data environments.

Most of the existing hashing methods for image retrieval are based on the assumption the image database is stationary. However, in the real world data environments are always changing or non-stationary, therefore the underlying data distribution may change from time to time which will result in the problem of concept drift. Incremental Hashing (ICH) is the only existing method to handle image retrieval with concept drift in non-stationary data environments. It builds hash codes for the database through increments. At each increment, a set of new hash functions is built with the new chunk of data, which is utilized to update the multi-hashing system to generate multiple sets of hash codes for all data. However, only the newest data chunk is used to train individual hash functions, while the semantic similarity information of previous data is missed. In this paper, we present a new hashing method based on ICH for image retrieval with concept drift, Incremental Hashing with Dynamic Semantic Pool (ICH-DSP). It builds a semantic pool to collect representative labeled data for each existing class. The semantic pool is updated incrementally and is used as the supervisory information for the training of hash functions. Experimental results on three real world image databases show that ICH-DSP outperforms the original ICH and other state-of-the-art hashing methods.

A very large volume of images is uploaded to the Internet daily. However, current hashing methods for image retrieval are designed for static databases only. They fail to consider the fact that the distribution of images can change when new images are added to the database over time. The changes in the distribution of images include both discovery of a new class and a distribution of images within a class owing to concept drift. Retraining of hash tables using all images in the database requires a large computation effort. This is also biased to old data owing to the huge volume of old images which leads to a poor retrieval performance over time. In this paper, we propose the incremental hashing (ICH) method to deal with the two aforementioned types of changes in the data distribution. The ICH uses a multihashing to retain knowledge coming from images arriving over time and a weight-based ranking to make the retrieval results adaptive to the new data environment. Experimental results show that the proposed method is effective in dealing with changes in the database.