Using attention mechanisms in saliency detection networks enables effective feature extraction, and using linear methods can promote proper feature fusion, as verified in numerous existing models. Current networks usually combine depth maps with red-green-blue (RGB) images for salient object detection (SOD). However, fully leveraging depth information complementary to RGB information by accurately highlighting salient objects deserves further study. We combine a gated attention mechanism and a linear fusion method to construct a dual-stream interactive recursive feature-reshaping network (IRFR-Net). The streams for RGB and depth data communicate through a backbone encoder to thoroughly extract complementary information. First, we design a context extraction module (CEM) to obtain low-level depth foreground information. Subsequently, the gated attention fusion module (GAFM) is applied to the RGB depth (RGB-D) information to obtain advantageous structural and spatial fusion features. Then, adjacent depth information is globally integrated to obtain complementary context features. We also introduce a weighted atrous spatial pyramid pooling (WASPP) module to extract the multiscale local information of depth features. Finally, global and local features are fused in a bottom-up scheme to effectively highlight salient objects. Comprehensive experiments on eight representative datasets demonstrate that the proposed IRFR-Net outperforms 11 state-of-the-art (SOTA) RGB-D approaches in various evaluation indicators.

Abstract Traditional pyramid pooling modules have shown effective improvements in semantic segmentation tasks by capturing multi-scale feature information. However, their limitations arise from the shallow structure, which fails to fully extract contextual information, and the fused multi-scale feature information lacks distinctiveness, resulting in issues with the final segmentation discriminability. To address these issues, we proposes an effective solution called FCPFNet, which is based on global contextual prior for deep feature extraction of detailed information. Specifically, we introduce a novel deep feature aggregation module to extract semantic information from the output feature map of each layer through a deep aggregation of context information module, and expands the effective perception range. Additionally, we propose an Efficient Pyramid Pooling Module (EPPM) to capture distinctive features through communicating information between different sub-features and performs multi-scale fusion, which is integrated as a branch within the network to complement the information loss resulting from downsampling operations. Furthermore, in order to ensure the richness of image detail feature information and maintain a large receptive field to obtain more contextual information, EPPM concatenates the input feature map and the output feature map of the pyramid pooling module to acquire more comprehensive global contextual information. It has been demonstrated by experiment that the method described in this article achieves competitive performance on the challenging scene segmentation datasets Pascal VOC 2012, Cityscapes and Coco-Stuff, with MIOU of 81.0%, 78.8% and 40.1%, respectively.

The performance of license plate detection has greatly improved with the development of deep learning. However, two challenges remain. First, in unconstrained scenarios, such as rotation and uneven lighting, license plate detection still faces significant challenges. Secondly, traditional horizontal bounding boxes are not suitable for representing multi-oriented license plates. And quadrilateral boxes can effectively represent them though, they cannot avoid the confusion caused by sequential label points. To address these challenges, we propose a license plate detection method called SvRetina-LPD. Specifically, we design an inception residual fusion pyramid network, which fuses multiple layers of features thoroughly. Then, by integrating spatial attention and channel attention, the inception residual multi-dimensional attention network is developed to weaken the noise and highlight the features of the license plate. Finally, we introduce a sliding vertex head network that incorporates a sliding vertex branch and regresses four length ratios to represent the relative sliding offsets of the corner points of the quadrilateral to the corresponding sides of the bounding rectangle. This method can accurately detect multi-directional license plate regions and effectively avoids the sequential label points. Extensive experimental tests were conducted on datasets such as CCPD, AOLP, and CLPD. On the CCPD test set, SvRetina-LPD demonstrated a high accuracy of 97.9%, surpassing existing methods. In addition, SvRetina-LPD also demonstrated excellent detection accuracy in other subsets of CCPD, including various challenging scenarios, demonstrating its ability to accurately detect multi-directional license plates in unconstrained scenarios.

Owing to the expansion in processing of industrial information through advances in machine learning, the demand for accurate crowd counting in various applications is increasing. We propose a multimodality cross-guided compensation coordination network (MC$^{3}$Net) for accurate red–green–blue and thermal (RGB-T) crowd counting. The network includes modules of intricate interactive fusion, feature difference compensation, and complementary attention enhancement. We use ConvNext as the backbone and process the three streams from RGB, thermal, and spliced RGB-T inputs. The multimodality data are sequentially guided and fused hierarchically, fully combining features extracted from the RGB and thermal images. Thereafter, difference compensation is applied to compress fusion and splicing features. Redundant information is removed. Then, feature mismatch is mitigated to enhance complementary information, reduce the loss of details, and finally obtain crowd statistics. Results from extensive experiments on the RGBT-CC dataset indicate the robustness and effectiveness of MC$^{3}$Net, which also achieves high performance on the DroneRGBT dataset and ShanghaiTechRGBD dataset, outperforming existing crowd counting methods. The code and models are available at: https://github.com/WBangG/MC3Net.

Fine-grained visual classification is more challenging as a subtask of image classification due to the large intra-class and slight inter-class variations. Recent work has focused on localizing discriminative features using attentional mechanisms. However, attention tends to focus on the salient parts of feature maps, ignoring other regions that are not salient but are discriminative for fine-grained classification. In this regard, we propose an efficient method called Discriminative Region Learning Dual-Branch Attention Network (DAL-Net) to address this problem. We propose: (1)Avoid over-focusing on local features by pixel-level attention wipe on salient features. (2)The features are enhanced and suppressed by the channel space enhancement module to mine multiple discriminative regions. (3)To learn complementary semantic information, we fuse cross-regional discriminative features from another branch. Our method can be trained end-to-end without bounding boxes and annotations. We demonstrate that our method can obtain competitive results on the CUB200-2011, FGVC-Aircraft, Stanford Cars, and Stanford Dogs datasets through comprehensive experiments.

Driven by the rapid development and application of earth observation sensors, the scene parsing of remote sensing images (RSIs) has attracted extensive research attention in recent years. Restricted by the limited local receptive field of successive convolution layers, traditional models of scene parsing cannot effectively and interactively utilize the local-global information and digital surface model (DSM) of RSIs. Comparatively, accurate scene parsing faces more challenges because of unbalanced categories, small targets, and more complex scenes. To address these challenges, herein, we propose a novel adjacent bi-hierarchical network (ABHNet). Specifically, we introduce a DSM-enhanced (DSE) module to excavate characteristic DSM information from DSM images and enhance the red, green, and blue (RGB) features by exploiting informative cues between RGB and DSM modalities. In addition, an adjacent context exploration (ACE) module is proposed, which contains current and adjacent branches. The branches first exploit multiscale complementary characteristics of multilevel features and then integrate these features by applying adjacent exploration. Our model includes five ACE modules—three are deployed to activate detailed features and two obtain deep-guided features. The mutual collaboration of deep and detailed features is more beneficial to the segmentation of small objects. Extensive experiments on two remote sensing benchmark datasets (ISPRS Potsdam and Vaihingen) showed that the proposed ABHNet qualitatively and quantitatively outperformed other methods.

Most recent methods for RGB (red-green-blue)-thermal salient object detection (SOD) involve several floating-point operations and have numerous parameters, resulting in slow inference, especially on common processors, and impeding their deployment on mobile devices for practical applications. To address these problems, we propose a lightweight spatial boosting network (LSNet) for efficient RGB-thermal SOD with a lightweight MobileNetV2 backbone to replace a conventional backbone (e.g., VGG, ResNet). To improve feature extraction using a lightweight backbone, we propose a boundary boosting algorithm that optimizes the predicted saliency maps and reduces information collapse in low-dimensional features. The algorithm generates boundary maps based on predicted saliency maps without incurring additional calculations or complexity. As multimodality processing is essential for high-performance SOD, we adopt attentive feature distillation and selection and propose semantic and geometric transfer learning to enhance the backbone without increasing the complexity during testing. Experimental results demonstrate that the proposed LSNet achieves state-of-the-art performance compared with 14 RGB-thermal SOD methods on three datasets while improving the numbers of floating-point operations (1.025G) and parameters (5.39M), model size (22.1 MB), and inference speed (9.95 fps for PyTorch, batch size of 1, and Intel i5-7500 processor; 93.53 fps for PyTorch, batch size of 1, and NVIDIA TITAN V graphics processor; 936.68 fps for PyTorch, batch size of 20, and graphics processor; 538.01 fps for TensorRT and batch size of 1; and 903.01 fps for TensorRT/FP16 and batch size of 1). The code and results can be found from the link of https://github.com/zyrant/LSNet.

The semantic segmentation of road scenes is an important task in autonomous driving. Deep learning has enabled the development of a variety of semantic segmentation networks using RGB and depth data. However, poor lighting conditions and long-distance sensing limit the applicability of RGB and depth cameras. Nevertheless, many existing methods still rely on precise depth maps for scene segmentation. Unlike depth information, thermal imaging provides a visual heat representation that remains accurate under a variety of lighting conditions and over longer distances. For robust and accurate segmentation of scenes collected during autonomous driving, we used the advanced MobileNetV2 network for feature extraction and a fusion strategy with an embedded control gate. In addition, we adopted an encoder–decoder scheme for semantic segmentation and developed an attention residual learning strategy to restore the resolution of the feature map. Finally, semantic and boundary supervision is introduced to optimize parameters of the proposed network. Experimental results show that the proposed network outperforms existing networks on segmentation of urban scenes, and our network can be generalized to depth data.

Scene parsing is a fundamental task in computer vision. Various RGB-D (color and depth) scene parsing methods based on fully convolutional networks have achieved excellent performance. However, color and depth information are different in nature and existing methods cannot optimize the cooperation of high-level and low-level information when aggregating modal information, which introduces noise or loss of key information in the aggregated features and generates inaccurate segmentation maps. The features extracted from the depth branch are weak because of the low quality of the depth map, which results in unsatisfactory feature representation. To address these drawbacks, we propose a progressive guided fusion and depth enhancement network (PGDENet) for RGB-D indoor scene parsing. First, high-quality RGB images are used to improve depth data through a depth enhancement module, in which the depth maps are strengthened in terms of channel and spatial correlations. Then, we integrate information from the RGB and enhance depth modalities using a progressive complementary fusion module, in which we start with high-level semantic information and move down layerwise to guide the fusion of adjacent layers while reducing hierarchy-based differences. Extensive experiments are conducted on two public indoor scene datasets, and the results show that the proposed PGDENet outperforms state-of-the-art methods in RGB-D scene parsing.

Understanding urban scenes is a fundamental capability required of intelligent vehicles. Depth cues provide useful geometric information for semantic segmentation, thus complementing RGB (color) data. Although single-modal RGB images are improved by depth information, semantic segmentation may be degraded in poor-visibility conditions. Thermal imaging can address some limitations of depth data. Therefore, we leverage the multimodal information in RGB-and-thermal (RGB-T) images by introducing a dynamic bilateral cross-fusion network (DBCNet) for RGB-T urban scene understanding. First, RGB-T features extracted by a given backbone are regrouped as high- or low-level features. Second, multimodal high-level features are sent to a dynamic bilateral cross-fusion module for further refinement. Third, a bounded high-level semantic-feature integration module is added to provide feature guidance, and a multitask supervision mechanism is used for fine-tuning. Extensive experiments on two RGB-T urban scene-understanding datasets indicate that DBCNet aggregates multilevel deep features effectively and outperforms state-of-the-art deep-learning scene-understanding methods.

Abstract Fine‐grained vehicle recognition is a challenging problem due to high inter‐class confusion among vehicle models under the influence of pose and viewpoint. To effectively describe the discriminative characteristics, many approaches try to learn detailed information from an individual image. Inspired by Siamese network that addresses the case where two inputs are relatively similar, the semantic interaction learning network (SIL‐Net) is designed to discover semantic differences between two fine‐grained categories via pairwise comparison. Specifically, SIL‐Net first collecting contrastive information by learning the mutual feature of input image pair, and then compare it with individual features to generate corresponding semantic features. These features learn semantic differences from contextual comparison, this gives SIL‐Net the ability to distinguish between two confusing images via pairwise interaction. After training, SIL‐Net can adaptively learn feature priorities under the supervision of the margin ranking loss and converge quickly. SIL‐Net performs well on two public vehicle benchmarks (Stanford Cars and CompCars), showing the suitability of SIL‐Net to fine‐grained vehicle recognition.

Scene parsing approaches have attracted extensive attention in recent years; although several methods have been developed for scene parsing, most include complex modules for both cross-modality fusion between RGB and depth images in the encoder and image scale level recovery in the decoder under label supervision for high inference accuracy. Cross-modality information in the encoder may be diluted when processed through the decoder, and the supervision results may not be reused effectively, which adversely affects scene parsing. To address these problems, we propose a recursive triple-path learning network (RTLNet) for cross-modality interactions in the decoder using global context and cross-modality fusion modules. The proposed modules fully use cross-modality information to reduce information loss. To enhance the robustness of RTLNet, we add a path to reuse the initial predictions from the decoder and introduce a ladder-shaped feature consistency module to further leverage multiscale features. Experiments are conducted with the proposed RTLNet and nine recent RGB-D indoor scene parsing methods on the NYUv2 and SUN-RGBD indoor scene datasets; the results show that the RTLNet outperforms the other methods.

Under ideal environmental conditions, RGB-based deep convolutional neural networks can achieve high performance for salient object detection (SOD). In scenes with cluttered backgrounds and many objects, depth maps have been combined with RGB images to better distinguish spatial positions and structures during SOD, achieving high accuracy. However, under low-light and uneven lighting conditions, RGB and depth information may be insufficient for detection. Thermal images are insensitive to lighting and weather conditions, being able to capture important objects even during nighttime. By combining thermal images and RGB images, we propose an effective and consistent feature fusion network (ECFFNet) for RGB-T SOD. In ECFFNet, an effective cross-modality fusion module fully fuses features of corresponding sizes from the RGB and thermal modalities. Then, a bilateral reversal fusion module performs bilateral fusion of foreground and background information, enabling the full extraction of salient object boundaries. Finally, a multilevel consistent fusion module combines features across different levels to obtain complementary information. Comprehensive experiments on three RGB-T SOD datasets show that the proposed ECFFNet outperforms 12 state-of-the-art methods under different evaluation indicators.

Scene parsing of high spatial resolution (HSR) remote sensing images has achieved notable progress in recent years by the adoption of convolutional neural networks. However, for scene parsing of multimodal remote sensing images, effectively integrating complementary information remains challenging. For instance, the decrease in feature map resolution through a neural network causes loss of spatial information, likely leading to blurred object boundaries and misclassification of small objects. In addition, object scales on a remote sensing image vary substantially, undermining the parsing performance. To solve these problems, we propose an end-to-end common extraction and gate fusion network (CEGFNet) to capture both high-level semantic features and low-level spatial details for scene parsing of remote sensing images. Specifically, we introduce a gate fusion module to extract complementary features from spectral data and digital surface model data. A gate mechanism removes redundant features in the data stream and extracts complementary features that improve multimodal feature fusion. In addition, a global context module and a multilayer aggregation decoder handle scale variations between objects and the loss of spatial details due to downsampling, respectively. The proposed CEGFNet was quantitatively evaluated on benchmark scene parsing datasets containing HSR remote sensing images, and it achieved state-of-the-art performance.

Most existing crowd counting approaches use limited information of RGB (red–green–blue) images and fail to suitably extract potential pedestrians in unconstrained scenarios. Moreover, complementary depth maps do not provide information of locations where people are more likely to be present. However, by incorporating optical and thermal information, the recognition of pedestrians may be enhanced considerably. In fact, thermal imaging information is robust to weather and lighting scenarios, and information from targets can be extracted even at nighttime. By combining RGB and thermal imaging information, we propose a dual-branch enhanced feature fusion network (DEFNet) for RGB-T (RGB and thermal) crowd counting. In DEFNet, an intensive data-enhancement module fuses complementary features of the same sizes from the RGB and thermal modalities, thus combining various rich receptive fields and generating powerful fused RGB-T features. These features describe both spatial structures and appearance details, highlighting information of crowd location. Then, an efficient dilation fusion module applies convolutions to the RGB -T features to obtain flexible and specific features, effectively eliminating the influence of background on the crowd information for density map prediction. Finally, high- and low-level features are used to efficiently obtain density maps through a fusion decoding module. Experimental results on an RGB-T crowd counting dataset indicate that the proposed DEFNet outperforms existing approaches. Furthermore, DEFNet can be generalized to handle RGB and depth data.

Compared with traditional handcrafted features, deep learning has greatly improved the performance of scene parsing. However, it remains challenging under various environmental conditions caused by imaging limitations. Thermal imaging cameras have several advantages over cameras for the visible spectrum, such as operation in total darkness, robustness to shadow effects, insensitivity to illumination variations, and strong ability to penetrate smog and haze. These advantages of thermal imaging cameras make them ideal for the scene parsing of semantic objects in daytime and nighttime. In this paper, we propose a novel multiscale feature fusion and enhancement network (MFFENet) for accurate parsing of RGB–thermal urban road scenes even when the quality of the available RGB data is compromised. The proposed MFFENet consists of two encoders, a feature fusion layer, and a multi-label supervision layer. We concatenate the multi-scale features with the features that contain global semantic information. Furthermore, we explore the cross-modal fusion of RGB and thermal features at multiple stages, rather than fusing them once at the low or high stage. Then, we propose a spatial attention mechanism module that provides a higher weight to (focuses more on) the foreground area, allowing MFFENet to emphasize foreground objects. Finally, multi-label supervision is introduced to optimize parameters of the proposed MFFENet. Experimental results confirm that the proposed MFFENet outperforms similar high-performing methods.

Owing to the widespread adoption of depth sensors, salient object detection (SOD) supported by depth maps for reliable complementary information is being increasingly investigated. Existing SOD models mainly exploit the relation between an RGB image and its corresponding depth information across three fusion domains: input RGB-D images, extracted feature maps, and output salient object. However, these models do not leverage the crossflows between high- and low-level information well. Moreover, the decoder in these models uses conventional convolution that involves several calculations. To further improve RGB-D SOD, we propose a crossflow and cross-scale adaptive fusion network (CCAFNet) to detect salient objects in RGB-D images. First, a channel fusion module allows for effective fusing depth and high-level RGB features. This module extracts accurate semantic information features from high-level RGB features. Meanwhile, a spatial fusion module combines low-level RGB and depth features with accurate boundaries and subsequently extracts detailed spatial information from low-level depth features. Finally, a purification loss is proposed to precisely learn the boundaries of salient objects and obtain additional details of the objects. The results of comprehensive experiments on seven common RGB-D SOD datasets indicate that the performance of the proposed CCAFNet is comparable to those of state-of-the-art RGB-D SOD models.

Summary Early and accurate assessment of plant disease severity is key to preventing disease attack. Traditional detection methods rely on manual vision to distinguish between types of disease infection, but this is time consuming, laborious and inaccurate. To address this problem, this paper proposes a deep learning‐based attentional network model (SEV‐Net) for plant disease severity identification and classification. The network embeds the improved channel and spatial attention module into the residual block of ResNet. The proposed attention module reduces the redundancy of information between channels and focuses on the most information‐rich regions of the feature map. In this experiment, SEV‐Net achieved an accuracy of 97.59% and 95.37% for multiple and single plant (Tomato) disease severity classification, which was better than existing attentional networks (SE‐Net and CBAM). Moreover, the combination of visualization techniques showed that SEV‐Net was adept at distinguishing small variations between plant diseases, proving the feasibility and effectiveness of the network. Furthermore, we have also designed and developed an Android application for real‐time classification of plant disease severity. The system deploys the SEV‐Net network model, which has higher classification accuracy and faster recognition speed.

Salient object detection (SOD) based on convolutional neural networks has achieved remarkable success. However, further improving the detection performance on challenging scenes (e.g., low-light scenes) requires additional investigation. Thermal infrared imaging captures thermal radiation from the surface of objects. Thus, it is insensitive to lighting conditions and can provide uniform imaging of objects. Accordingly, we propose a two-stage fusion network (TSFNet) integrating RGB and thermal information for RGB-T SOD. For the first fusion stage, we propose a feature-wise fusion module that captures and aggregates united information and intersecting information in each local region of the RGB and thermal images, and then independent decoding is applied to the RGB and thermal features. For the second fusion stage, we propose a bilateral auxiliary fusion module that extracts auxiliary spatial features from the foreground and background of the thermal and RGB modalities. Finally, we use multiple supervision to further improve the SOD performance. Comprehensive experiments demonstrate that TSFNet outperforms 11 state-of-the-art models under various indicators on three RGB-T SOD datasets.

Exploiting RGB and depth information can boost the performance of semantic segmentation. However, owing to the differences between RGB images and the corresponding depth maps, such multimodal information should be effectively used and combined. Most existing methods use the same fusion strategy to explore multilevel complementary information at various levels, likely ignoring different feature contributions at various levels for segmentation. To address this problem, we propose a network using a two-stage cascaded decoder (TCD), embedding a detail polishing module, to effectively integrate high- and low-level features and suppress noise from low-level details. Additionally, we introduce a depth filter and fusion module to extract informative regions from depth cues with the guidance of RGB images. The proposed TCD network achieves comparable performance to state-of-the-art RGB-D semantic segmentation methods on the benchmark NYUDv2 and SUN RGB-D datasets.

The use of RGB-D information for salient object detection (SOD) is being increasingly explored. Traditional multilevel models handle both low- and high-level features similarly, as they use the same number of features for blending. Unlike these models, in this paper, we propose multilevel reverse-context interactive-fusion (MRI) network (MRINet) for RGB-D SOD. Specifically, first, we extract and reuse different numbers of features depending on their level; the deeper the information, the more times do we perform the extraction. Deeper information contains more semantic cues, which are important for locating salient regions. Thereafter, we use an RGB MRI block (MRIB) to merge RGB information at different levels; furthermore, we use depth features as auxiliary information and an RGB-D MRIB for full merging with RGB information. RGB and RGB-D MRIBs can reconstruct the high-level feature map in high resolution and integrate the low-level feature map to enhance boundary details. Extensive experiments demonstrate the effectiveness of the proposed MRINet and its state-of-the-art performance in RGB-D SOD.

Semantic segmentation is a fundamental task in computer vision, and it has various applications in fields such as robotic sensing, video surveillance, and autonomous driving. A major research topic in urban road semantic segmentation is the proper integration and use of cross-modal information for fusion. Here, we attempt to leverage inherent multimodal information and acquire graded features to develop a novel multilabel-learning network for RGB-thermal urban scene semantic segmentation. Specifically, we propose a strategy for graded-feature extraction to split multilevel features into junior, intermediate, and senior levels. Then, we integrate RGB and thermal modalities with two distinct fusion modules, namely a shallow feature fusion module and deep feature fusion module for junior and senior features. Finally, we use multilabel supervision to optimize the network in terms of semantic, binary, and boundary characteristics. Experimental results confirm that the proposed architecture, the graded-feature multilabel-learning network, outperforms state-of-the-art methods for urban scene semantic segmentation, and it can be generalized to depth data.

In recent years, the detection of distinctive objects in stereoscopic 3D images has drawn increasing attention. Unlike 2D salient object detection, salient object detection in stereoscopic 3D images is highly challenging. Hence, we propose a novel Deep Convolutional Residual Autoencoder (DCRA) for end-to-end salient object detection in stereoscopic 3D images. The core trainable architecture of the salient object detection model employs raw stereoscopic 3D images as the inputs and their corresponding ground truth saliency masks as the labels. A convolutional residual module is applied to both the encoder and the decoder as a basic building block in the DCRA, and long-range skip connections are employed to bypass the equal-sized feature maps between the encoder and the decoder. To explore the complex relationships and exploit the complementarity between RGB (photometric) and depth (geometric) information, multiple feature map fusion modules are constructed. These modules integrate texture and structure information between the RGB and depth branches of the encoder and fuse their features over several multiscale layers. Finally, to efficiently optimize DCRA parameters, a supervision pyramid based on boundary loss and background prior loss is adopted, which employs supervised learning over the multiscale layers in the decoder to prevent vanishing gradients and accelerate the training at the fusion stage. We compare the proposed DCRA with state-of-the-art methods on two challenging benchmark datasets. The results of these experiments demonstrate that our proposed DCRA performs favorably against the comparison models.

Many RGB-D visual attention models have been proposed with diverse fusion models; thus, the main challenge lies in the differences in the results between the different models. To address this challenge, we propose a local-global fusion model for fixation prediction on an RGB-D image; this method combines global and local information through a content-aware fusion module (CAFM) structure. First, it comprises a channel-based upsampling block for exploiting global contextual information and scaling up this information to the same resolution as the input. Second, our Deconv block contains a contrast feature module to utilize multilevel local features stage-by-stage for superior local feature representation. The experimental results demonstrate that the proposed model exhibits competitive performance on two databases.

Most RGB-D salient object detection (SOD) models use the same network to process RGB images and their corresponding depth maps. Subsequently, these models perform direct concatenation and summation at deep or shallow layers. However, these models ignore the complementarity of multi-level features extracted from RGB images and depth maps. This paper presents an asymmetric deeply fused network (ADFNet) for RGB-D SOD. Two different backbone networks, i.e., ResNet-50 and VGG-16, are utilized to process RGB images and related depth maps. We use an aggregation decoder and adaptive attention transformer module (AATM) to avoid information loss in the decoding process. Additionally, we use an attention early fusion module (AEFM) and deep fusion module (DFM) to deal with the deep features in various complex situations. Experiments validate the effectiveness of the proposed ADFNet, which outperforms thirteen recent RGB-D SOD models in the analysis of five public RGB-D SOD datasets.

Summary Attribute reduction is one of the most important topics in rough set theory. In the classical rough sets, the method for attribute reduction is mainly to keep positive region, boundary region, and negative region unchanged. However, the three regions are no longer monotonic with respect to adding or deleting an attribute in decision‐theoretic rough sets. In decision‐theoretic rough set model, the decision regions are determined by using the Bayesian decision procedure, and decision‐making should take consideration of the cost. In this paper, two attribute reduction methods based on minimum decision cost are proposed from the algebraic view and the information theory, respectively. First, significance of joint attributes is introduced to measure the classification ability of selected attribute subset to decision‐making, which overcomes the disadvantage of only considering the significance of single attribute. By using significance of joint attributes, a heuristic method based on minimum decision cost for attribute reduction is presented. Second, conditional mutual information is proposed to evaluate the significance of attribute subset for decision‐making in minimum cost attribute reduction. To decrease the computational complexity of the conditional mutual information, an approximate computation method is calculated from both maximum relevance and maximum significance. To evaluate the two proposed algorithms, extensive experiments are conducted on 10 University of California at Irvine data sets. We compare our proposed algorithms with several existing cost minimization attribute reduction algorithms. Experiment results show that our proposed algorithms have a superior performance in achieving the reduct. Copyright © 2016 John Wiley & Sons, Ltd.

Abstract The growing peak‐hour power demand has invoked an urgency to increase the peak‐hour supply. Although smart grid has been envisioned as the next generation power system due to its two‐way communication of information and power, the peak‐hour power shortage problem still exists. In this paper, we propose a credit‐based incentive tariff (CIT) scheme with fraud‐traceability for smart grid. Specifically, the CIT encourages retail customers to sell the power generated by their renewable resources back to the grid during peak hours via giving additional incentive rate to them based on their credits. If a fraud is detected during the power transaction, the malicious customer's identity can be traced out and his or her credit can be correspondingly reduced. The security analysis shows that the CIT resists various security threats and makes the incentive tariff fair and more secure. The performance evaluation demonstrates that the CIT can dramatically increase the peak‐hour supply and reduce the peak‐to‐average power demand ratio by up to 7%. Copyright © 2013 John Wiley & Sons, Ltd.

Unsupervised hyperspectral band selection has been an important topic in hyperspectral imagery. This technique aims at selecting some critical and decisive spectral bands from an original image for compact representation without compromising and distorting the raw information in the relevant spectral bands. Although many efforts have been made to this topic, the structural information has not yet been well exploited during band selection, and there are still several deficiencies in search strategies, leaving room for further improvement. This paper tackles the unsupervised hyperspectral band selection problem from a global perspective and proposes a novel method claiming the following main contributions: structure-aware measures for band informativeness and independence; and a graph formulation of band selection allowing for an efficient integrated search by means of dominant set extraction. Experiments on three real hyperspectral images demonstrate the superiority of the proposed band selector in comparison with benchmark methods.

This paper tackles a recent challenge in identifying culprit actors, who try to hide confidential payload with steganography, among many innocent actors in social media networks. The problem is called steganographer detection problem and is significantly different from the traditional stego detection problem that classifies an individual object as a cover or a stego. To solve the steganographer detection problem over large-scale social media networks, this paper proposes a method that uses high-order joint features and clustering ensembles. It employs 250-D features calculated from the high-order joint matrices of Discrete Cosine Transform (DCT) coefficients of JPEG images, which indicate the dependencies of image content. Furthermore, a number of hierarchical sub-clusterings trained by the features are integrated as a clustering ensemble based on the majority voting strategy, which is used to make optimal decisions on suspicious steganographers. Experimental results show that the proposed scheme is effective and efficient in identifying potential steganographers in large-scale social media networks, and has better performance when tested against the state-of-the-art steganographic methods.

Non-intrusive appliance load monitoring (NIALM) helps identify major energy guzzlers in a building without introducing extra metering cost. It motivates users to take proper actions for energy saving and greatly facilitates demand response (DR) programs. Nevertheless, NIALM of large-scale appliances is still an open challenge. To pursue a scalable solution to energy monitoring for contemporary large-scale appliance groups, we propose a distributed metering platform and use parallel optimization for semi-intrusive appliance load monitoring (SIALM). Based on a simple power model, a sparse switching event recovering (SSER) model is established to recover appliance states from their aggregated load data. Furthermore, the sufficient conditions for unambiguous state recovery of multiple appliances are presented. By considering these conditions as well as the electrical network topology constraint, a minimum number of meters are obtained to correctly recover the energy consumption of individual appliances. We evaluate the performance of both SIALM and NIALM with real-world trace data and synthetic data. The results demonstrate that with the help of a small number of meters, the SIALM approach significantly improves the accuracy of energy disaggregation for large-scale appliances.

With the increasing sensing and communication in cyber physical social system (CPSS), the data volume is growing much rapidly in recent years. Secure deduplication has attracted considerable interests of storage provider for data management efficiency and data privacy preserving. One of the most challenging issues in secure deduplication is how to manage data and the convergent key when users frequently update it. To solve this problem, D. Koo et al. use bilinear paring as the key method. However, bilinear paring requires high computation cost for implementations. In this paper, we propose a session-key-based convergent key management scheme, named SKC, to secure the dynamic update in the data deduplication. Specifically, each data owner in SKC can verify the correctness of the session key and dynamically change it with the data update. Furthermore, to enable group combination and remove the aid of gateway (GW), a convergent key sharing scheme, named CKS, is presented. Security analysis demonstrates that both SKC and CKS can protect the confidentiality of the data and the convergent key in the case of dynamic updates. The simulation results show that our SKC and CKS can significantly reduce computation complexity and communication during the data uploading phase.

Time synchronization is one of the most fundamental services for wireless sensor networks. Prior studies have investigated the clock stability due to environmental dynamics. In this paper, we demonstrate by experiment that in spite of the surrounding environment, time synchronization is unavoidably impacted by in-network energy heterogeneity, which may incur up to 30-40 ppm clock uncertainty. We mathematically analyze the root cause of such clock uncertainty and propose a protocol called EATS. Sensor nodes with EATS can intelligently select the best synchronization parents that minimize the negative impact of the energy heterogeneity. The selection is robust to multiple impacting factors in the network and provides fine-grained synchronization accuracy. In addition, nodes can make use of local energy information and further calibrate the clocks. In light of this, the logic time maintained among different nodes is more consistent and the synchronization can be performed with a longer re-synchronization interval and less energy consumption. We implement EATS with TelosB motes and evaluate the effectiveness and efficiency of our design through extensive experiments and simulations.

In this work, we address the problem of distributed expert finding using chains of social referrals and profile matching with only local information in online social networks. By assuming that users are selfish, rational, and have privately known cost of participating in the referrals, we design a novel truthful efficient mechanism in which an expert-finding query will be relayed by intermediate users. When receiving a referral request, a participant will locally choose among her neighbors some user to relay the request. In our mechanism, several closely coupled methods are carefully designed to improve the performance of distributed search, including, profile matching, social acquaintance prediction, score function for locally choosing relay neighbors, and budget estimation. We conduct extensive experiments on several data sets of online social networks. The extensive study of our mechanism shows that the success rate of our mechanism is about 90 percent in finding closely matched experts using only local search and limited budget, which significantly improves the previously best rate 20 percent. The overall cost of finding an expert by our truthful mechanism is about 20 percent of the untruthful methods, e.g., the method that always selects high-degree neighbors. The median length of social referral chains is 6 using our localized search decision, which surprisingly matches the well-known small-world phenomenon of global social structures.

Load curve data in power systems refers to users' electrical energy consumption data periodically collected with meters. It has become one of the most important assets for modern power systems. Many operational decisions are made based on the information discovered in the data. Load curve data, however, usually suffers from corruptions caused by various factors, such as data transmission errors or malfunctioning meters. To solve the problem, tremendous research efforts have been made on load curve data cleansing. Most existing approaches apply outlier detection methods from the supply side (i.e., electricity service providers), which may only have aggregated load data. In this paper, we propose to seek aid from the demand side (i.e., electricity service users). With the help of readily available knowledge on consumers' appliances, we present an appliance-driven approach to load curve data cleansing. This approach utilizes data generation rules and a Sequential Local Optimization Algorithm (SLOA) to solve the Corrupted Data Identification Problem (CDIP). We evaluate the performance of SLOA with real-world trace data and synthetic data. The results indicate that, comparing to existing load data cleansing methods, such as B-spline smoothing, our approach has an overall better performance and can effectively identify consecutive corrupted data. Experimental results also show that our method is robust in various tests.

Distributed energy resources DERs, which are characterized by small-scale power generation technologies to provide an enhancement of the traditional power system, have been strongly encouraged to be integrated into the smart grid, and numerous trading strategies have recently been proposed to support the energy auction in the emerging smart grid marketing. However, few of them consider the security aspects of energy trading, such as privacy preservation, bid integrity, and pre-filtering ability. In this paper, we propose an efficient searchable encryption scheme for auction SESA in emerging smart grid marketing. Specifically, SESA uses a public key encryption with keyword search technique to enable the energy sellers e.g., DERs to inquire suitable bids while preserving the privacy of the energy buyers. Additionally, to facilitate the seller to search for detailed information of the bids, we also propose an extension of SESA to support conjunctive keywords search. Security analysis demonstrates that the proposed SESA and its extension can achieve data and keyword privacy, bid integrity and trapdoor unforgeability. Simulation results also show that both SESA and its extension have less computation and communication overhead than the existing searchable encryption approaches. Copyright © 2013 John Wiley & Sons, Ltd.

Based on plenty of sensors on smartphones and tablets, their location information becomes more accurate and easy to access. Many position-based applications have thus been developed. As these mobile devices have become the main access approach of 802.11-based wireless local area networks (WLAN), location information also provides opportunities to improve the performance of underlying wireless communication. This paper presents CO-MAP (Co-Occurrence MAP), which leverages location information to handle exposed and hidden terminal problems, which are still a major cause of throughput degradation in mobile WLANs. With the positions of nodes, CO-MAP rapidly builds a co-occurrence map showing which two links can occur concurrently. Meanwhile, to avoid potential collisions, it selects the best settings of frame transmissions based on a novel analytic network model when hidden terminals are distinguished. CO-MAP improves the goodputs of both downlinks and uplinks in an instant and distributed manner. Our implementation on a hardware testbed demonstrates that CO-MAP can accurately detect potential interferers, and provide significant gain of goodput for both exposed and hidden terminal scenarios. The simulation results also suggest that imperfect position hints can still bring substantial improvement on the multiple access efficiency.

With the deployment of smart meters at individual households, smart grid can collect metering data of users' power consumption. However, users' power usage patterns would also be revealed. To preserve the users' privacy, metering data is mostly encrypted by cryptographic algorithms. When data mining is needed to support decision making or ensure reliability, to find useful information from the encrypted data is very important for smart grid. Most of the traditional keyword searching schemes rarely consider both users' data privacy and requesters' query privacy. In particular, the power system data in smart grid has multidimensional attributes; thus, how to query over the encrypted multidimensional data on all dimensions is a challenging issue in smart grid. To achieve finer grained conjunctive query, this paper proposes an Efficient Conjunctive Query (ECQ) scheme. Specificly, the ECQ incorporates the idea of public key encryption and conjunctive keywords search to achieve conjunctive query without data and query privacy leakage. Security analysis demonstrates that the ECQ can achieve the security requirements, namely, data confidentiality, integrity and privacy, as well as query privacy. In addition, simulation results show that the ECQ can reduce users' computation cost and total communication cost.

With the development of the Internet, user-generated data has been growing tremendously in Web 2.0 era. Facing such a big volume of resources in folksonomy, people need a method of fast exploration and indexing to find their demanded data. To achieve this goal, contextual information is indispensable and valuable to understand user preference and purpose. In sociolinguistics, context can be mainly categorized as verbal context and social context. Comparing with verbal context, social context not only requires domain knowledge to pre-define contextual attributes but also acquires additional data from users. However, there is no research of addressing irrelevant contextual factors for verbal context model so far. The dominating set from verbal context proposed in this paper is to fill this blank. We present the verbal context in folksonomy to capture the user intention, and propose a dominating set discovering method for this verbal context model to prune the irrelevant contextual factors and keep the major characteristics at the same time. Furthermore, the experiments, which are conducted on a public data set, show that the proposed method gives convincing results.

Infrared and visible image fusion, which highlights radiometric and detailed texture information and completely and accurately describes objects, is a long-standing and well-studied task in computer vision. Existing convolutional neural network-based approaches that leverage end-to-end networks to fuse infrared and visible images have made significant progress. However, most approaches typically extract the features in the encoder segment and use a coarse fusion strategy. Unlike these algorithms, this study proposes a multiscale receptive field amplification fusion network (MRANet) to effectively extract the local and global features from images. Particularly, we extract long-range information in the encoder segment using a convolutional residual structure as the main backbone and a simplified uniformer as an auxiliary backbone, both of which are ResNet-inspired. Additionally, we propose an effective multiscale fusion strategy based on an attention mechanism to integrate the two modalities. Extensive experiments demonstrate that MRANet performs efficiently on image fusion datasets.