Non-transferable learning (NTL) has emerged as a promising method to protect the intellectual property of deep learning models by restricting cross-domain knowledge transfer. However, the robustness of its transferability constraints against potential attacks has not been explored, especially in practical deployment scenarios. In this paper, we propose a novel black-box attack framework - distribution drift learner (DDL), which effectively bypasses NTL protection mechanisms by only accessing input-output queries of protected models. The theoretical foundation of DDL is derived from the concept of data drift, which takes advantage of the variability of the statistical distribution between the source and target domains. The core innovation of DDL is the integration of distributed perception regularization into a lightweight autoencoder architecture, enabling efficient manipulation of data distribution by optimizing dual objectives (distributed perception loss and reconstruction loss). Training for DDL involves two key steps: First, DDL reconstructs a moderate amount of target domain samples and feeds the reconstructed images into the NTL model to obtain prediction labels. The DDL parameters are then updated by optimizing distributed perception loss and reconstruction loss. Through extensive experiments against standard NTL benchmarks (Digits, CIFAR10, and STL10), we demonstrate that DDL has successfully overcome the barriers of the transferable NTL model and improved the accuracy of the target domain by 81% from 10%. Our work reveals critical vulnerabilities in the NTL framework, particularly with respect to ownership verification and applicability authorization mechanisms, providing valuable insights for developing more robust model protection strategies in real-world applications.

Deep neural network (DNN) models embody the intellectual property of a model owner, as the process of training the DNN model is a complex and resource-intensive task that requires significant investments in data preparation and computing resources. Numerous efforts have been made to protect the intellectual property of DNN models. However, existing methods often come with a critical limitation: they lack robustness, proving effective only in specific intellectual property threat scenarios or they either sacrifice the utility/accuracy of the model owner’s classifier because it interferes with the classifier’s training. To address these issues, we propose GMFIP, a novel generator-based model fingerprinting technology tailored for DNN intellectual property protection. GMFIP stands out for its robustness, extending its utility to various intellectual property threat scenarios rather than specific ones. Furthermore, GMFIP ensures that the utility/accuracy of the model is not affected by protection measures. Specifically, GMFIP begins with the training of the generator, which lays the groundwork for the model fingerprint. The generator generates fingerprints of the unique properties of the source model for verifying model ownership. To further improve the quality of these fingerprints, an extra selection phase dedicated to refining the fingerprints is integrated. Moreover, GMFIP is complemented by a binary classifier, which adapts the threshold setting to get optimal results. Our empirical evaluation includes an ablation study over four state-of-the-art technologies and three image benchmark datasets. Our results demonstrate that GMFIP outperforms other state-of-the-art technologies in effectively distinguishing pirated models from benign models.

The accurate extraction of traffic signs is of great significance to the digitization of traffic information and the fine management of traffic. This article introduces an innovative approach to address the challenges associated with recognizing and detecting traffic signs, considering their vulnerability to complex backgrounds, variations in illumination, and motion blur. The proposed method utilizes a semi-supervised learning (SSL) strategy, combining convolutional neural networks (CNNs) with a transformer encoder–decoder architecture, to extract traffic sign features from vehicle panoramic images. To enhance feature extraction, a hierarchical sampling method (HSM) is introduced, which facilitates the extraction of multiscale self-attention features in the transformer encoder–decoder structure. Additionally, a network module called local and global information aggregator (LGIA) is designed based on HSM, enabling the incorporation of both local and global context information. Furthermore, a SSL strategy is adopted to simultaneously train our model using both labeled and unlabeled data samples. This strategy aims to improve the extraction of traffic signs by capitalizing on the broader data set available through unlabeled data. Experimental results demonstrate the effectiveness and robustness of the proposed method in improving the detection and recognition of traffic signs. The approach showcases significant improvements in overcoming the challenges posed by complex backgrounds, variations in illumination, and motion blur. Our approach achieved a 0.9% improvement in the F1-score evaluation over the current classical object detection algorithm on the public data set Tsinghua-Tencent 100K and a 1.1% improvement on the SSW data set.

The success of machine learning (ML) depends on the availability of large-scale datasets. However, recent studies have shown that models trained on such datasets are vulnerable to privacy attacks, among which membership inference attack (MIA) brings serious privacy risk. MIA allows an adversary to infer whether a sample belongs to the training dataset of the target model or not. Though a variety of defenses against MIA have been proposed such as differential privacy and adversarial regularization, they also result in lower model accuracy and thus make the models less unusable. In this article, aiming at maintaining the accuracy while protecting the privacy against MIA, we propose a new defense against membership inference attacks by generative adversarial network (GAN). Specifically, sensitive data is used to train a GAN, then the GAN generate the data for training the actual model. To ensure that the model trained with GAN on small datasets can has high utility, two different GAN structures with special training techniques are utilized to deal with the image data and table data, respectively. Experiment results show that the defense is more effective on different data sets against the existing attack schemes, and is more efficient compared with most advanced MIA defenses.

Accurate and automatic detection of road surface element (such as road marking or manhole cover) information is the basis and key to many applications. To efficiently obtain the information of road surface element, we propose a content-adaptive hierarchical deep learning model to detect arbitrary-oriented road surface elements from mobile laser scanning (MLS) point clouds. In the model, we design a densely connected feature integration module (DCFM) to connect and reorganize feature maps of each stage in the backbone network. Besides, we propose a hierarchical prediction module (HPM) to innovatively use the reorganized feature maps to recognize different types of road surface elements, and thus, semantic information of road surface element can be adaptively expressed on multilevel feature maps. We also add a cascade structure (CS) in the head of model to detect the target efficiently, which can learn the offset between the predicted minimum bounding box of road surface element and ground truth. In experiments, we prove that the proposed method mainly contributed by HPM can maintain robust detection performance, even in the cases of unbalanced category number or overlapping of road surface elements. The experiments also prove that the proposed DCFM can improve the recognition effects of small targets. The CS for predicting boundary offset can detect each target more accurately. We also integrate the designed modules into some rotation detectors, e.g., the EAST and R3Det, and achieve the state-of-the-art results in three road scenes with different categories and uneven distribution of road surface elements, which further shows the effectiveness of the proposed method.

In recent years, point clouds have been widely used in powerline inspection, smart cities, autonomous driving, and other fields. Deep learning-based point cloud processing methods have achieved some impressive results in point cloud semantic segmentation, which has attracted more and more attention. However, there are still some problems that need to be solved, such as the efficiency of point cloud processing, inference speed, the parameter size of network, etc. We mainly study how to remove the redundancy of neural network for large-scale point cloud semantic segmentation. Now that the scale of point cloud dataset has rapidly increased, many recent works adapt to it via expanding the model capacity, which can lead to a sharp decline in the efficiency and speed of processing point cloud. To address the problem, we propose an efficient and lightweight deep neural network, namely Slight Filter Learning Network (SFL-Net), which can effectively extract semantic information and accelerate semantic segmentation for large-scale point clouds. The key to our approach is the proposed Slight Filter Convolution module (SFConv) and the Hourglass Block (HB). SFConv is designed to remove the redundancy of 3D convolution filters. HB can replace all multilayer perceptron (MLPs) in the neural network to expedite point cloud processing. To reduce the information loss during the subspace transformation, we introduce a new correlation loss function to constrain the parameter pairs in HB. On public indoor and outdoor datasets evaluation, SFL-Net performance reaches and even outperforms the state-of-the-art approaches. Moreover, the model parameters of SFL-Net are reduced 10× than the KPConv. The inference time of SFL-Net is only 45.8s for about 100 million points (N~108).

Over the past few years, a variety of membership inference attacks against deep learning models have emerged, raising significant privacy concerns. These attacks can easily infer whether a sample exists in the training set of the target model with little adversary knowledge, and the inference accuracy is often much higher than random guessing, which causes serious privacy leakage. To this end, defenses against membership inference attacks have attracted great interest. However, the current available defense methods such as regularization, differential privacy, and knowledge distillation are unable to balance the trade-off between privacy and utility well. In this paper, we combine knowledge distillation and generative adversarial networks to propose a novel training framework that can effectively defend against membership inference attacks, called KD-GAN. Extensive experiments show that our method implements an attack success rate of nearly 0.5 (random guesses) which can successfully defend against membership inference attacks without causing significant damage to model utility, and consistently outperforming other defense methods in the balance of privacy and utility.

In recent years, point clouds have been widely used in powerline inspection, smart cities, autonomous driving, and other fields. The deep learning-based point cloud processing methods have attracted more and more attention due to the developments of laser scanning technology and machine learning. However, the recent methods largely ignore global contextual relationships and do not make full use of the complementation between local feature and high-level geometric information. To the problem, we propose a novel deep neural network, namely, the Dense connection-based Kernel Point Network (DenseKPNet), which can greatly expand the receptive field of kernel point convolution to extract rich semantic context information and valuable geometric features from the local region effectively. Specifically, we first design a multiscale convolution kernel point module to extract initial geometric features from coarse to fine. Then, we design the dense connection module to efficiently learn more expressive local geometric features while capturing rich contextual information. In addition, we propose the kernel point convolution attention module (KPCAM), which can capture global interdependencies between points and strengthen the discriminativeness of effective features. We evaluate our method on public indoor and outdoor datasets. The qualitative and quantitative experimental results show the effectiveness of DenseKPNet. The mIoU of the proposed method on S3DIS and semantic3D datasets can reach 68.9% and 77.9%, respectively.

The maintenance of subway tunnels is vital to ensure the safety of their daily operation. Issues experienced by shield subway tunnels, especially the water leakages, require rapid and accurate detection and diagnosis. Due to the large number of disturbances in the tunnels, conventional algorithms face limitations when extracting discriminative features. To solve this problem, we propose a novel and efficient deep learning model for extracting multiscale and discriminative features of water leakages based on mobile laser scanning (MLS) point cloud intensity images. A new residual network module (Res2Net) is integrated with a cascade structure to form a unified model to extract the multiscale features of water leakages. The model can fully consider geometric characteristics of water leakages and grade the residual connections in a single residual block. This expands the size of receptive field in each network layer and can better facilitate the extraction of geometric characteristics of water leakages. Finally, we verify the advantages of the proposed method via experiments on five water leakage datasets of tunnel intensity images converted from point clouds obtained by a self-developed MLS system and compare its performance with other methods.

The point cloud registration is important and necessary for the applications of changing detection, deformation monitoring, and so on, which is also challenging due to the vast clustered points, irregular, and complex structures of spatial objects, and quality effects of the labeled corresponding points. Aiming at this problem, we design an end-to-end 3-D graph deep learning framework of point cloud registration, which can simultaneously learn the detector (graph attention expression) and the descriptor (graph deep feature) for point cloud registration in a weakly supervised way, so that the learned detector and descriptor promote each other in the process of model optimization. Then, the detector is used to automatically extract the keypoints, and the descriptor describes the deep feature of each keypoint. In the framework, we innovatively propose a new module (named MLP_GCN), which fuses multilayer perceptron (MLP) and graph convolutional network (GCN). The MLP_GCN module is further integrated into the detector branch and descriptor branch to fully express the detector and descriptor of the point cloud. In the training process of the framework, we rotate and translate the point cloud randomly to form the training data in a weakly supervised way, which can save plenty of manually labeling time of corresponding points. In the experiments, our method can achieve better results of point cloud registration in comparison with other methods, which verifies the advantages of the proposed method.

Adversarial attack is an important research direction of neural network security, and the black-box attack in the unknown model is an important application scenario of adversarial attack. The adversarial samples generated by existing black-box attack algorithms often have drastic hue changes, are easy to be perceived by human eyes, and the algorithm computational efficiency is generally low. Therefore, this paper proposes a multi-scale grid search for the image gray attack strategy, performing perturbations of different intensity on image areas of different scales, and then makes the generated sample hue change more natural. Experiments show that the performance of the method is better than the baseline algorithm, and greatly improves the attack efficiency, increasing the neural network query number over 50% comparing with the fastest baseline algorithm.

Metro subway systems with underground tunnels form the backbone of urban transportations and therefore, accurate monitoring and maintenance of such subway systems are extremely necessary for a hassle-free daily commutation of billions of people. Though 3-D models of tunnels are widely used for the deformation monitoring of such subway tunnels, existing model-based tunnel monitoring systems rely on coarse geometric models and hence fail to capture complete tunnel health information. We present a two-stage algorithm to create high-fidelity geometric models of tunnel lining from Terrestrial Laser Scanning (TLS) point clouds. Tunnel geometry, defined at the detailed block entity level, is constructed through a data-driven block segmentation algorithm and a model-driven assembly technique. In our approach, the 3-D tunnel block segmentation problem has been translated into a bolt and lining joint recognition problem from 2-D images unfolded from the 3-D scans. The segmented 3-D blocks are matched with a set of predefined 3-D templates from a primitive library via a constraint total least squares matching method and the matched 3-D templates are assembled to create the final watertight tunnel model. The proposed tunnel modeling method has been comprehensively evaluated on Changzhou, Nanjing, and Wuhan tunnel data sets in terms of outliers, missing data, point density, topological representation, robustness, and geometric accuracy. The experiments on Nanjing and Changzhou metro tunnels show that the geometric model fitting incurs an error of only 7 mm, which is almost consistent with a mean density of 6 mm of these two data sets. Experimental results validate the advantages and potentials of the proposed tunnel modeling method.

Accurate and efficient extraction of road marking plays an important role in road transportation engineering, automotive vision, and automatic driving. In this article, we proposed a dense feature pyramid network (DFPN)-based deep learning model, by considering the particularity and complexity of road marking. The DFPN concatenated its shallow feature channels with deep feature channels so that the shallow feature maps with high resolution and abundant image details can utilize the deep features. Thus, the DFPN can learn hierarchical deep detailed features. The designed deep learning model was trained end to end for road marking instance extraction with mobile laser scanning (MLS) point clouds. Then, we introduced the focal loss function into the optimization of deep learning model in road marking segmentation part, to pay more attention to the hard-classified samples with a large extent of background. In the experiments, our method can achieve better results than state-of-the-art methods on instance segmentation of road markings, which illustrated the advantage of the proposed method.

Classification of airborne laser scanning (ALS) point clouds is needed in digital cities and 3-D modeling. To efficiently recognize objects in ALS point clouds, we propose a novel hierarchical aggregated deep feature representation method, which can adequately employ spatial association of multilevel structures and deep feature discrimination. In our method, a 3-D deep learning model is constructed to represent the discriminative feature of each point cluster in a hierarchical structure by decreasing the within-class distance and increasing the between-class distance. Our method aggregates the discriminative deep features in different levels into a hierarchical aggregated deep feature that considers the spatial hierarchy and feature distinctiveness. Lastly, we build a multichannel 1-D convolutional neural network to classify the unknown points. Our tests demonstrate that the proposed hierarchical aggregated deep feature method can enhance point cloud classification results. Comparing with seven state-of-the-art methods, those results also verified the superior performance of our method.

The 4-points congruent sets (4PCS) techniques have widely been used for global registration of point clouds in terrestrial laser scanning (TLS) applications. Nevertheless, due to many 4PCS methods adopt a downsampling strategy in the collection of correspondences; it is challenging to obtain a real congruent set of tuples in different point clouds with varying density. This letter embeds multiscale sparse features (MSSF) into 4PCS to enable efficient global registration of TLS clouds. Specifically, multiscale clusters are used to extract point features, among which a sparse coding is performed to obtain the representative MSSF. The obtained MSSF are then embedded into the 4PCS variants, taking both geometrical structure and representative feature similarity into account when performing a four-point congruent tuples matching. Moreover, a normal constraint is considered in the selection of noncoplanar four-point bases rather than coplanar ones in the source cloud. This configuration decreases the number of four-point bases and thus improves the processing efficiency and registration accuracy. The proposed method was applied to two experiments with TLS point clouds from building and hillslope scenarios, respectively. In addition, the proposed MSSF configuration was embedded in two 4PCS variants, and compared with the original methods for global registration. The comparison shows evident improvements in terms of the registration accuracy and efficiency if our embedment is used.

Due to errors in sensors and positioning, there exist mismatches between different phases of mobile laser scanning point clouds, which impede the application of point cloud, such as changing detection and deformation monitoring. To rectify such mismatches, we designed a 3-D deep feature construction method for point cloud registration. The proposed method combines two 3-D convolutional neural networks into a uniform deep learning model to extract 3-D deep features. First, the corresponding points and noncorresponding points are set to train the deep learning model to minimize the distance between corresponding points’ features and maximize the distance between features of noncorresponding points. Second, in the test phase, the 3-D deep feature for each keypoint was extracted by the trained deep learning model. This could be used to determine the corresponding points by the$k$-dimensional tree and random sample consensus (RANSAC) algorithm. Finally, a transformation matrix was calculated based on the corresponding points and was then applied to point cloud registration. The experimental results illustrated that the proposed method of using 3-D deep features is more efficient at a corresponding point search than representatives of three existing methods. It also improved registration accuracy.

To efficiently recognize on-ground objects in airborne laser scanning (ALS) point clouds, we design a method that jointly learns a discriminative dictionary and a classifier. In the method, the point cloud is segmented into hierarchical point clusters, which are organized by a tree structure. Then, the feature of each point cluster is extracted. The feature of a leaf node is obtained by aggregating the features of all its parent nodes. The feature of the leaf node is called the hierarchical aggregation feature. The hierarchical aggregation features are encoded by sparse coding. We introduce a new label consistency constraint called “discriminative sparse-code error,” and combine it with the reconstruction error, the classification error, and L1-norm sparsity constraint to form a unified objective function. The objective function is efficiently solved by using the proposed label consistency feature sign method. We obtain an overcomplete discriminative dictionary and an optimal linear classifier. Experiments performed on different ALS point cloud scenes have shown that the hierarchical aggregation features combined with the learned classifier can significantly enhance the classification results, and also demonstrated the superior performance of our method over other techniques in point cloud classification.

The ability to classify urban objects in large urban scenes from point clouds efficiently and accurately still remains a challenging task today. A new methodology for the effective and accurate classification of terrestrial laser scanning (TLS) point clouds is presented in this paper. First, in order to efficiently obtain the complementary characteristics of each 3-D point, a set of point-based descriptors for recognizing urban point clouds is constructed. This includes the 3-D geometry captured using the spin-image descriptor computed on three different scales, the mean RGB colors of the point in the camera images, the LAB values of that mean RGB, and the normal at each 3-D point. The initial 3-D labeling of the categories in urban environments is generated by utilizing a linear support vector machine classifier on the descriptors. These initial classification results are then first globally optimized by the multilabel graph-cut approach. These results are further refined automatically by a local optimization approach based upon the object-oriented decision tree that uses weak priors among urban categories which significantly improves the final classification accuracy. The proposed method has been validated on three urban TLS point clouds, and the experimental results demonstrate that it outperforms the state-of-the-art method in classification accuracy for buildings, trees, pedestrians, and cars.

With the emergence of huge volumes of high-resolution remote sensing images produced by all sorts of satellites and airborne sensors, processing and analysis of these images require effective retrieval techniques. To alleviate the dramatic variation of the retrieval accuracy among queries caused by the single image feature algorithms, we developed a novel graph-based learning method for effectively retrieving remote sensing images. The method utilizes a three-layer framework that integrates the strengths of query expansion and fusion of holistic and local features. In the first layer, two retrieval image sets are obtained by, respectively, using the retrieval methods based on holistic and local features, and the top-ranked and common images from both of the top candidate lists subsequently form graph anchors. In the second layer, the graph anchors as an expansion query retrieve six image sets from the image database using each individual feature. In the third layer, the images in the six image sets are evaluated for generating positive and negative data, and SimpleMKL is applied to learn suitable query-dependent fusion weights for achieving the final image retrieval result. Extensive experiments were performed on the UC Merced Land Use-Land Cover data set. The source code has been available at our website. Compared with other related methods, the retrieval precision is significantly enhanced without sacrificing the scalability of our approach.

Efficient presentation and recognition of on-ground objects from airborne laser scanning (ALS) point clouds are a challenging task. In this paper, we propose an approach that combines a discriminative-dictionary-learning-based sparse coding and latent Dirichlet allocation (LDA) to generate multilevel point-cluster features for ALS point-cloud classification. Our method takes advantage of the labels of training data and each dictionary item to enforce discriminability in sparse coding during the dictionary learning process and more accurately further represent point-cluster features. The multipath AdaBoost classifiers with the hierarchical point-cluster features are trained, and we apply them to the classification of unknown points by the heritance of the recognition results under different paths. Experiments are performed on different ALS point clouds; the experimental results have shown that the extracted point-cluster features combined with the multipath classifiers can significantly enhance the classification accuracy, and they have demonstrated the superior performance of our method over other techniques in point-cloud classification.

Point cloud classification plays a critical role in point cloud processing and analysis. Accurately classifying objects on the ground in urban environments from airborne laser scanning (ALS) point clouds is a challenge because of their large variety, complex geometries, and visual appearances. In this paper, a novel framework is presented for effectively extracting the shape features of objects from an ALS point cloud, and then, it is used to classify large and small objects in a point cloud. In the framework, the point cloud is split into hierarchical clusters of different sizes based on a natural exponential function threshold. Then, to take advantage of hierarchical point cluster correlations, latent Dirichlet allocation and sparse coding are jointly performed to extract and encode the shape features of the multilevel point clusters. The features at different levels are used to capture information on the shapes of objects of different sizes. This way, robust and discriminative shape features of the objects can be identified, and thus, the precision of the classification is significantly improved, particularly for small objects.

Triangular Irregular Network (TIN), especially Constrained Delaunay Triangular Irregular Network (CD-TIN), has been proven to be more accurate for the modeling of complex urban terrain, compared with raster represented digital elevation model (DEM). DEM with high precision is of importance to simulate urban flood inundation. In the paper, the triangular-prism sets are introduced to calculate the inundation water surface within each basin. It is worth mentioning that the explored dichotomy numerical solution is efficient on the simulation of the inundation depth. The case of ramp-shape urban terrain validates the correctness of the algorithm. And the application of the “7.21” storm in main campus of Beijing Normal University (BNU) verifies the usability and practicality of the algorithm. The cases demonstrate the feasibility of the proposed algorithm for simulation of urban flood inundation associated with urban surface runoff and drainage system capacity information.