Neural Ordinary Differential Equations (Neural ODEs or NODEs) excel at modeling continuous dynamical systems from observational data, especially when the data is irregularly sampled. However, existing training methods predominantly rely on numerical ODE solvers, which are time-consuming and prone to accumulating numerical errors over time due to autoregression. In this work, we propose VF-NODE, a novel approach based on the variational formulation (VF) to accelerate the training of NODEs. Unlike existing training methods, the proposed VF-NODEs implement a series of global integrals, thus evaluating Deep Neural Network (DNN)--based vector fields only at specific observed data points. This strategy drastically reduces the number of function evaluations (NFEs). Moreover, our method eliminates the use of autoregression, thereby reducing error accumulations for modeling dynamical systems. Nevertheless, the VF loss introduces oscillatory terms into the integrals when using the Fourier basis. We incorporate Filon's method to address this issue. To further enhance the performance for noisy and incomplete data, we employ the natural cubic spline regression to estimate a closed-form approximation. We provide a fundamental analysis of how our approach minimizes computational costs. Extensive experiments demonstrate that our approach accelerates NODE training by 10 to 1000 times compared to existing NODE-based methods, while achieving higher or comparable accuracy in dynamical systems. The code is available at https://github.com/ZhaoHongjue/VF-NODE-ICLR2025.

Reinforcement Learning (RL) has made significant strides in various domains, and policy gradient methods like Proximal Policy Optimization (PPO) have gained popularity due to their balance in performance, training stability, and computational efficiency. These methods directly optimize policies through gradient-based updates. However, developing effective control policies for environments with complex and non-linear dynamics remains a challenge. High variance in gradient estimates and non-convex optimization landscapes often lead to unstable learning trajectories. Koopman Operator Theory has emerged as a powerful framework for studying non-linear systems through an infinite-dimensional linear operator that acts on a higher-dimensional space of measurement functions. In contrast with their non-linear counterparts, linear systems are simpler, more predictable, and easier to analyze. In this paper, we present Koopman-Inspired Proximal Policy Optimization (KIPPO), which learns an approximately linear latent-space representation of the underlying system’s dynamics while retaining essential features for effective policy learning. This is achieved through a Koopman-approximation auxiliary network that can be added to the baseline policy optimization algorithms without altering the architecture of the core policy or value function. Extensive experimental results demonstrate consistent improvements over the PPO baseline with 6–60% increased performance while reducing variability by up to 91% when evaluated on various continuous control tasks.

This research focuses on improving the fine-tuning process of self-supervised learning models for remote sensing, particularly the Cross-Scale Masked Auto-Encoder (MAE). We tackle the challenges of intricate, multi-source imagery and present advancements in adapting the Cross-Scale MAE for diverse remote sensing environments. Our contributions include methods for handling complex dataset dimensions and semantic diversity, demonstrating the model’s adaptability and expanding its application scope in remote sensing.

In terms of monitoring and managing anthropogenic activities, accurately identifying the distinct phases in construction projects using satellite imagery remains a challenging task. In this paper, we reformulate the phase classification problem into a transition detection problem and introduce a novel Koopman-based Transition Detection (KTD) method, which applies Koopman operator theory to analyze the nonlinear dynamics of material histograms in a linear framework. KTD employs a sliding window to perform Dynamic Mode Decomposition (DMD) on the time-series material histograms and detects the transition point by analyzing the movement of the eigenvalue in consecutive strides. Compared to CNN-based methods, our proposed KTD method demonstrates enhanced accuracy and reduced temporal error in phase identification. Furthermore, as an unsupervised method that does not require large amounts of training data, it shows a better generalization capability in the sequestered region.

Despite recent progress in static image segmentation, video segmentation is still challenging due to the need for an accurate, fast, and temporally consistent model. Conducting per-frame static image segmentation on a video is not acceptable since it is computationally prohibitive and prone to temporal inconsistency. In this paper, we present bidirectional occlusion-guided feature propagation (BOFP) method with the goal of improving temporal consistency of segmentation results without sacrificing segmentation accuracy, while at the same time keeping the operations at a low computation cost. It leverages temporal coherence in the video by feature propagation from keyframes to other frames along the motion paths in both forward and backward directions. We propose an occlusion-based attention network to estimate the distorted areas based on bidirectional optical flows, and utilize them as cues for correcting and fusing the propagated features. Extensive experiments on benchmark datasets demonstrate that the proposed BOFP method achieves superior performance in terms of temporal consistency while maintaining comparable level of segmentation accuracy at a low computation cost, striking a great balance among the three performance metrics essential to evaluate video segmentation solutions.

False data injection attacks (FDIAs) pose a significant security threat to power system state estimation. To detect such attacks, recent studies have proposed machine learning (ML) techniques, particularly deep neural networks (DNNs). However, most of these methods fail to account for the risk posed by adversarial measurements, which can compromise the reliability of DNNs in various ML applications. In this paper, we present a DNN-based FDIA detection approach that is resilient to adversarial attacks. We first analyze several adversarial defense mechanisms used in computer vision and show their inherent limitations in FDIA detection. We then propose an adversarial-resilient DNN detection framework for FDIA that incorporates random input padding in both the training and inference phases. Our simulations, based on an IEEE standard power system, demonstrate that this framework significantly reduces the effectiveness of adversarial attacks while having a negligible impact on the DNNs' detection performance. Index Terms-False Data Injection Attack, Smart Grid Communication, Deep Learning, Adversarial Attacks

Deep Learning Object Detection relies on extensive, manual annotation of datasets, a time-consuming and costly process prone to human inconsistencies. Auto-labeling using Visual Foundation Models offers a promising alternative but often falls short in object detection tasks. This research introduces a novel framework that uses the Segment Anything Model (SAM) with minimal annotated images to create an effective object detector. Despite the capabilities of Visual Foundation Models in downstream tasks, our research reveals their poor performance in object detection when operating within a different domain. Additionally, we demonstrate that with only a few labeled images, we can create a much better and simpler object detection system. We also prove that our model outperforms the best existing object detectors when it comes to analyzing drone images taken in pineapple fields.

With the ever-increasing proliferation of edge devices for applications, such as home automation and vehicle systems, their security vulnerabilities have received additional attention. A recent type of attack, physical fault injections, are particularly powerful as they can compromise these devices by skipping necessary instructions through physical methods, such as induced voltage glitches. Hence, they can trigger a wide range of software behavior anomalies and vulnerabilities not caused by the programs themselves. These attacks allow adversaries to carry out severe security breaches such as control flow hijacking and information leakage, even if the original device firmware has been well tested.

Despite the great progress in hyperspectral image classification, it is still challenging due to the unique characteristic of satellite imagery where the training and test sets may come from different distributions because of the different acquisition conditions. Hence, directly deploying the trained model on the test data may lead to degradation in the performance. In this work, we propose an unsupervised domain adaptation approach that aligns distributions across the training and test domains. It projects the data to a shared embedding space, i.e., the abundance space, that is regularized by physical constraints. The shared abundance space, together with a metricbased distribution alignment approach applied on the abundance space, would largely reduce the domain discrepancy and provide a more representative feature set for classification purpose. Experimental results on hyperspectral benchmarks demonstrate superiority of the proposed method.

Remote sensing images present unique challenges to image analysis due to the extensive geographic coverage, hardware limitations, and misaligned multi-scale images. This paper revisits the classical multi-scale representation learning prob- lem but under the general framework of self-supervised learning for remote sensing image understanding. We present Cross-Scale MAE, a self-supervised model built upon the Masked Auto-Encoder (MAE). During pre-training, Cross-Scale MAE employs scale augmentation techniques and enforces cross-scale consistency constraints through both contrastive and generative losses to ensure consistent and meaningful representations well-suited for a wide range of downstream tasks. Further, our implementation leverages the xFormers library to accelerate network pre-training on a single GPU while maintaining the quality of learned represen- tations. Experimental evaluations demonstrate that Cross-Scale MAE exhibits superior performance compared to standard MAE and other state-of-the-art remote sensing MAE methods.

Semantic segmentation in large-scale aerial images is an extremely challenging task. On one hand, the limited ground truth, as compared to the vast area the images cover, greatly hinders the development of supervised representation learning. On the other hand, the large footprint from remote sensing raises new challenges for semantic segmentation. In addition, the complex and ever changing image acquisition conditions further complicate the problem where domain shifting commonly occurs. In this paper, we exploit self-supervised contrastive learning (CL) methodologies for semantic segmentation in aerial imagery. In addition to performing CL at the feature level as most practices do, we add another level of contrastive learning, at the semantic level, taking advantage of the segmentation output from the downstream task. Further, we embed local mutual information in the semantic-level CL to enforce local consistency. This has largely enhanced the representation power at each pixel and improved the generalization capacity of the trained model. We refer to the proposed approach as multi-level contrastive learning with local consistency (mCL-LC). The experimental results on different benchmarks indicate that the proposed mCL-LC exhibits superior performance as compared to other state-of-the-art contrastive learning frameworks for the semantic segmentation task. mCL-LC also carries better generalization capacity especially when domain shifting exists.

Domain adaptation techniques have been widely applied to the problem of cross-scene hyperspectral image (HSI) classification. Most existing methods use convolutional neural networks (CNNs) to extract statistical features from data and often neglect the potential topological structure information between different land cover classes. CNN-based approaches generally only model the local spatial relationships of the samples, which largely limits their ability to capture the nonlocal topological relationship that would better represent the underlying data structure of HSI. In order to make up for the above shortcomings, a Topological structure and Semantic information Transfer network (TSTnet) is developed. The method employs the graph structure to characterize topological relationships and the graph convolutional network (GCN) that is good at processing for cross-scene HSI classification. In the proposed TSTnet, graph optimal transmission (GOT) is used to align topological relationships to assist distribution alignment between the source domain and the target domain based on the maximum mean difference (MMD). Furthermore, subgraphs from the source domain and the target domain are dynamically constructed based on CNN features to take advantage of the discriminative capacity of CNN models that, in turn, improve the robustness of classification. In addition, to better characterize the correlation between distribution alignment and topological relationship alignment, a consistency constraint is enforced to integrate the output of CNN and GCN. Experimental results on three cross-scene HSI datasets demonstrate that the proposed TSTnet performs significantly better than some state-of-the-art domain-adaptive approaches. The codes will be available from the website: https://github.com/YuxiangZhang-BIT/IEEE_TNNLS_TSTnet.

In this work, we propose a novel online knowledge distillation (OKD) approach, built upon the classical deep mutual learning framework in which peer networks (students) treat each other as teachers by learning from their predictions. The proposed method traces and leverages two levels of information encoded in each peer's learning trajectory to dynamically construct superior teachers to supervise other students. We first build a recurrent neural network associated with each peer, which takes both the network's current and previous logits as input and outputs integrated logits with the same dimension as the transferred knowledge. By doing so, the teachers provide an enhanced representation of knowledge. Beyond that, we also build a weight-averaged surrogate for each network, which maintains the exponential moving average of its learned parameters during the online training procedure. The proposed approach exploits the hidden information behind the online learning process instead of myopically learning from peers' outputs at a single time/iteration step. It potentially reduces uncertainties from peers as suffered in previous OKD studies with more stabilized transferred knowledge. We evaluate the proposed approach with benchmark image classification datasets and network architectures. Experimental results demonstrate its effectiveness with clear performance improvement over state-of-the-arts.

Online knowledge distillation (KD) mutually trains a group of student networks from scratch in a peer-teaching manner, eliminating the need for pre-trained teacher models. However, supervision from peers can be noisy, especially in the early stage of training. In this paper, we propose a novel method for online knowledge distillation by temporal-spatial boosting (TSB). The proposed method constructs superior "teachers" with two modules, temporal accumulator and spatial integrator. Specifically, the temporal accumulator leverages the previous outputs of networks during training and produces a representative prediction over all classes. Instead of merely imitating the outputs of other networks as in vanilla online KD, we further propose the so-called spatial integrator that consolidates the knowledge learned by all networks and yields a stronger instructor. The operations of these two modules are simple and straightforward, which can be computed efficiently on the fly during training. The proposed method can improve the efficiency of transferring effective knowledge as well as stabilize the training process. Experimental results on various benchmark datasets and network structures validate the effectiveness of the proposed method over the state-of-the-art.

ingle image depth estimation is an ill-posed problem. That is, it is not mathematically possible to uniquely estimate the 3rd dimension (or depth) from a single 2D image. Hence, additional constraints need to be incorporated in order to regulate the solution space. In this paper, we explore the idea of constraining the model by taking advantage of the similarity between the RGB image and the corresponding depth map at the geometric edges of the 3D scene for more accurate depth estimation. We propose a general light-weight adaptive geometric attention module that uses the cross-correlation between the encoder and the decoder as a measure of this similarity. More precisely, we use the cosine similarity between the local embedded features in the encoder and the decoder at each spatial point. The proposed module along with the encoder-decoder network is trained in an end-to-end fashion and achieves superior and competitive performance in comparison with other state-of-the-art methods. In addition, adding our module to the base encoder-decoder model adds only an additional 0.03% (or 0.0003) parameters. Therefore, this module can be added to any base encoder-decoder network without changing its structure to address any task at hand.

Effective and efficient animal disease detection and control have drawn increasing attention in smart farming in recent years. It is crucial to explore how to harvest data and enable data-driven decision making for rapid diagnosis and early treatment of infectious diseases among herds. This article proposes an IoT-based animal social behavior sensing framework to model mastitis propagation and infer mastitis infection risks among dairy cows. To monitor cow social behaviors, we deploy portable GPS devices on cows to track their movement trajectories and contacts with each other. Based on those collected location data, we build directed and weighted cattle social behavior graphs by treating cows as vertices and their contacts as edges, assigning contact frequencies between cows as edge weights, and determining edge directions according to contact spatial-temporal information. Then, we propose a flexible probabilistic disease transmission model, which considers both direct contacts with infected cows and indirect contacts via environmental contamination, to estimate and forecast mastitis infection probabilities. Our model can answer two common questions in animal disease detection and control: 1) which cows should be given the highest priorities for an investigation to determine whether there are already infected cows on the farm and 2) how to rank cows for further screening when only a tiny number of sick cows have been identified. Both theoretical and simulation-based analytics of in-the-field experiments (17 cows and more than 70-h data) demonstrate the proposed framework’s effectiveness. In addition, somatic cell count (SCC) mastitis tests validate our predictions as correct in real-world scenarios.

Photorealistic stylization aims to transfer the style of a reference photo onto a content photo in a natural fashion, such that the stylized image looks like a real photo taken by a camera. State-of-the-art methods stylize the image locally within each matched semantic region and are prone to global color inconsistency across semantic objects/parts, making the stylized image less photorealistic. To tackle the challenging issues, we propose a non-local representation scheme, constrained with a mutual affine-transfer network (NL-MAT). Through a dictionary-based decomposition, NL-MAT is able to successfully decouple matched non-local representations and color information of the image pair, such that the context correspondence between the image pair is incorporated naturally, which largely facilitates local style transfer in a global-consistent fashion. To the best of our knowledge, this is the first attempt to address the photorealistic stylization problem with a non-local representation scheme, such that no additional models or steps for semantic matching are required during stylization. Experimental results demonstrate that, the proposed method is able to generate photorealistic results with local style transfer while preserving both the spatial structure and global color consistency of the content image.

Hyperspectral images (HSIs) provide rich spectral information that has contributed to the successful performance improvement of numerous computer vision and remote sensing tasks. However, it can only be achieved at the expense of images’ spatial resolution. HSI super-resolution (HSI-SR), thus, addresses this problem by fusing low-resolution (LR) HSI with the multispectral image (MSI) carrying much higher spatial resolution (HR). Existing HSI-SR approaches require the LR HSI and HR MSI to be well registered, and the reconstruction accuracy of the HR HSI relies heavily on the registration accuracy of different modalities. In this article, we propose an unregistered and unsupervised mutual Dirichlet-Net ($u^{2}$-MDN) to exploit the uncharted problem domain of HSI-SRwithout the requirement of multimodality registration. The success of this endeavor would largely facilitate the deployment of HSI-SR since registration requirement is difficult to satisfy in real-world sensing devices. The novelty of this work is threefold. First, to stabilize the fusion procedure of two unregistered modalities, the network is designed to extract spatial information and spectral information of two modalities with different dimensions through a shared encoder–decoder structure. Second, the mutual information (MI) is further adopted to capture the nonlinear statistical dependencies between the representations from two modalities (carrying spatial information) and their raw inputs. By maximizing the MI, spatial correlations between different modalities can be well characterized to further reduce the spectral distortion. We assume that the representations follow a similar Dirichlet distribution for their inherent sum-to-one and nonnegative properties. Third, a collaborative$l_{2,1}$-norm is employed as the reconstruction error instead of the more common$l_{2}$-norm to better preserve the spectral information. Extensive experimental results demonstrate the superior performance of$u^{2}$-MDN as compared to the state of the art.

Given the prior information of the target, hyperspectral target detection focuses on exploiting spectral differences to separate objects of interest from the background, which can be treated as information retrieval (IR) task in machine learning (ML). Most traditional detection methods work in the original feature space and rely heavily on specific assumptions, which cannot guarantee effective extraction of features for the target and background in hyperspectral images (HSIs). Mass estimation (ME) is a base modeling mechanism that has been proven to effectively solve problems in IR and is not restricted by specific assumptions. In this article, we propose a novel target detection method through ensemble-based IR with ME (EIRME). By directly deriving the ordering from a sample set to rank data points, ME provides a simple and straightforward ranking measure to ensure that points similar to the given target are far away from dissimilar points. For the estimation of mass distribution, the proposed method utilizes a tree-structured mapping to generate a feature space, in which the separability of the target and background is further improved. In particular, to break through the technical difficulty that the direct migration of IR methods with mass measure cannot specifically meet the high-precision requirements of target detection in HSIs, we develop a specialized measurement, topological mass, which innovatively combines the mass measure with tree topology to quantify the spectral difference for detection output. Moreover, the IR with ME based on parallel measurements through ensemble trees provides a robust solution with better generalization capacity and higher precision for hyperspectral target detection, facilitating practical applications. Experimental results on benchmark HSI datasets prove that the specialized measurement that we developed successfully overcomes the drawbacks of the direct migration of IR methods with ME and exhibits unique advantages. In addition, comparisons with the most classic and advanced detection algorithms demonstrate the superiority of the proposed method.

Incentive mechanisms are essential for stimulating adequate worker participation to achieve good truth discovery performance in mobile crowdsensing (MCS) systems. However, most of existing incentive mechanisms only consider compensating workers’ sensing cost, while the cost incurred by potential privacy leakage has been largely neglected. Moreover, none of existing privacy-preserving incentive mechanisms has incorporated workers’ different privacy preferences to provide personalized payments for them. In this paper, we propose a contract-based personalized privacy-preserving incentive mechanism for truth discovery in MCS systems, named Paris-TD, which provides personalized payments for workers as a compensation for privacy cost while achieving accurate truth discovery. The basic idea is that the platform offers a set of different contracts to workers with different privacy preferences, and each worker chooses to sign a contract which specifies a privacy-preserving degree (PPD) and the corresponding payment the worker will receive if she submits perturbed data with that PPD. Specifically, we respectively design a set of optimal contracts analytically under both full and incomplete information models, which maximize the truth discovery accuracy under a given budget, while satisfying the individual rationality and incentive compatibility properties. The feasibility and effectiveness of Paris-TD are validated through experiments on both synthetic and real-world datasets.

Recent research demonstrated that the superficially well-trained machine learning (ML) models are highly vulnerable to adversarial examples. As ML techniques are becoming a popular solution for cyber-physical systems (CPSs) applications in research literatures, the security of these applications is of concern. However, current studies on adversarial machine learning (AML) mainly focus on pure cyberspace domains. The risks the adversarial examples can bring to the CPS applications have not been well investigated. In particular, due to the distributed property of data sources and the inherent physical constraints imposed by CPSs, the widely-used threat models and the state-of-the-art AML algorithms in previous cyberspace research become infeasible.

In this work, we demonstrate the challenges in predicting HPC cluster power consumption in the face of significant temporal skew in power consumption behavioral patterns. Predicting large power swings that extend several megawatts has significant operational value for HPC centers, however, prediction is challenging due to the relative rarity of such events and also due to the abrupt or disjoint deviation from the average power consumption levels. To study the impact of this challenge, we have trained a recurrent neural network (RNN) as a reasonably sophisticated model to predict power consumption of the one-year worth of node power consumption data from the Summit supercomputer located in the Oak Ridge Leadership Computing Facility. By studying the prediction results, we have found that although simple usage of RNN models can provide good results on average power consumption levels, it would fail at predicting the power swings that have more operational value. With such results, we discuss potential next steps in addressing such issues aiming towards a robust usage of power prediction techniques in HPC operations.

The perception system in autonomous vehicles is responsible for detecting and tracking the surrounding objects. This is usually done by taking advantage of several sensing modalities to increase robustness and accuracy, which makes sensor fusion a crucial part of the perception system. In this paper, we focus on the problem of radar and camera sensor fusion and propose a middle-fusion approach to exploit both radar and camera data for 3D object detection. Our approach, called CenterFusion, first uses a center point detection network to detect objects by identifying their center points on the image. It then solves the key data association problem using a novel frustum-based method to associate the radar detections to their corresponding object's center point. The associated radar detections are used to generate radar-based feature maps to complement the image features, and regress to object properties such as depth, rotation and velocity. We evaluate CenterFusion on the challenging nuScenes dataset, where it improves the overall nuScenes Detection Score (NDS) of the state-of-the-art camera-based algorithm by more than 12%. We further show that CenterFusion significantly improves the velocity estimation accuracy without using any additional temporal information. The code is available at https://github.com/mrnabati/CenterFusion.

Hyperspectral image (HSI) classification is one of the most active research topics and has achieved promising results boosted by the recent development of deep learning. However, most state-of-the-art approaches tend to perform poorly when the training and testing images are on different domains, e.g., the source domain and target domain, respectively, due to the spectral variability caused by different acquisition conditions. Transfer learning-based methods address this problem by pretraining in the source domain and fine-tuning on the target domain. Nonetheless, a considerable amount of data on the target domain has to be labeled and nonnegligible computational resources are required to retrain the whole network. In this article, we propose a new transfer learning scheme to bridge the gap between the source and target domains by projecting the HSI data from the source and target domains into a shared abundance space based on their own physical characteristics. In this way, the domain discrepancy would be largely reduced such that the model trained on the source domain could be applied to the target domain without extra efforts for data labeling or network retraining. The proposed method is referred to as physically constrained transfer learning through shared abundance space (PCTL-SAS). Extensive experimental results demonstrate the superiority of the proposed method as compared to the state of the art. The success of this endeavor would largely facilitate the deployment of HSI classification for real-world sensing scenarios.

Pansharpening is to fuse a multispectral image (MSI) of low-spatial-resolution (LR) but rich spectral characteristics with a panchromatic image (PAN) of high spatial resolution (HR) but poor spectral characteristics. Traditional methods usually inject the extracted high-frequency details from PAN into the upsampled MSI. Recent deep learning endeavors are mostly supervised assuming that the HR MSI is available, which is unrealistic especially for satellite images. Nonetheless, these methods could not fully exploit the rich spectral characteristics in the MSI. Due to the wide existence of mixed pixels in satellite images where each pixel tends to cover more than one constituent material, pansharpening at the subpixel level becomes essential. In this article, we propose an unsupervised pansharpening (UP) method in a deep-learning framework to address the abovementioned challenges based on the self-attention mechanism (SAM), referred to as UP-SAM. The contribution of this article is threefold. First, the SAM is proposed where the spatial varying detail extraction and injection functions are estimated according to the attention representations indicating spectral characteristics of the MSI with subpixel accuracy. Second, such attention representations are derived from mixed pixels with the proposed stacked attention network powered with a stick-breaking structure to meet the physical constraints of mixed pixel formulations. Third, the detail extraction and injection functions are spatial varying based on the attention representations, which largely improves the reconstruction accuracy. Extensive experimental results demonstrate that the proposed approach is able to reconstruct sharper MSI of different types, with more details and less spectral distortion compared with the state-of-the-art.

Hyperspectral unmixing is an important problem for remotely sensed data interpretation. It amounts at estimating the spectral signatures of the pure spectral constituents in the scene (endmembers) and their corresponding subpixel fractional abundances. Although the unmixing problem is inherently nonlinear (due to multiple scattering), the nonlinear unmixing of hyperspectral data has been a very challenging problem. This is because nonlinear models require detailed knowledge about the physical interactions between the sunlight scattered by multiple materials. In turn, bilinear mixture models (BMMs) can reach good accuracy with a relatively simple model for scattering. In this article, we develop a new BMM and a corresponding unsupervised unmixing approach which consists of two main steps. In the first step, a deep autoencoder is used to linearly estimate the endmember signatures and their associated abundance fractions. The second step refines the initial (linear) estimates using a bilinear model, in which another deep autoencoder (with a low-rank assumption) is adapted to model second-order scattering interactions. It should be noted that in our developed BMM model, the two deep autoencoders are trained in a mutually interdependent manner under the multitask learning framework, and the relative reconstruction error is used as the stopping criterion. The effectiveness of the proposed method is evaluated using both synthetic and real hyperspectral data sets. Our experimental results indicate that the proposed approach can reasonably estimate the nature of nonlinear interactions in real scenarios. Compared with other state-of-the-art unmixing algorithms, the proposed approach demonstrates very competitive performance.

Target detection aims to locate targets of interest within a specific scene. The traditional model-driven detectors based on signal processing have proved to be very effective. However, the detection performance of such traditional methods relies heavily on the model assumption, which is limited by the discrepancy with real hyperspectral images (HSIs) data. In this article, a target detection method through tree-structured encoding (TD-TSE) for HSIs is proposed. Instead of modeling the target and the background to extract valid features, we construct a binary tree based on the features of the data itself and segment the HSI to improve the separability of the target and the background. For the purpose of highlighting the target and suppressing the background, a novel measurement of separation, distance on tree, is calculated via binary encoding based on the constructed tree structure, and the detection output can be obtained according to such distance. To further reduce the generalization error resulting from random subsampling, the statistical average of the distances on multiple independent trees is estimated to improve the robustness of TD-TSE. The proposed method is not constrained by any model assumptions, which is fundamentally different from the most widely used hyperspectral target detectors in the field of signal processing. Moreover, the construction of binary trees without any labeled samples and the linear complexity of the proposed method make it highly practical for the hyperspectral data in real scenes. Extensive experiments on three benchmark HSI data sets demonstrate the effectiveness of the proposed TD-TSE for hyperspectral target detection.

Network pruning has achieved significant success in compressing and accelerating CNNs. However, the existing three-step iterative pipeline, which includes ranking, pruning, and fine-tuning, is extremely computationally expensive due to the feed-forward and back-propagation operations conducted in both the ranking and fine-tuning steps. In this paper, we present a computationally efficient framework for structured pruning by exploring the potential of leveraging the intermediate results generated during the fine-tuning step to rank the importance of filters and thus converting a three-step pipeline (with precise ranking) to a two-step pipeline (with coarse ranking), resulting in significant savings in computation time while achieving comparable performance in terms of classification accuracy as compared to the state-of-the-art. The proposed method is evaluated with various benchmark architectures and datasets for the image classification task. Experimental results show that the proposed approach can achieve superior performance in computation efficiency while maintaining the same accuracy level. Our approach would largely facilitate pruning practice, especially on resource-constrained platforms.

Hyperspectral nonlinear unmixing (HNU) is an extremely challenging problem as it is very difficult, if possible at all, to derive an explicit model to describe the underlying nonlinear mixing process. This paper gives the first attempt to tackle this problem by taking advantage of recent advances in deep learning, in specific, the development in generative adversarial network (GAN). The biggest contribution of GAN is that upon training, the network can generate samples with the same probabilistic distribution as that of the training samples, without explicitly knowing what the distribution actually is. Hence, we ask a similar question: can we unmix a hyperspectal image without explicitly knowing the nonlinear mixing model? In order to test this hypothesis, this paper proposes a data-driven supervised HNU method as compared to the traditional model-based approaches and uses a specific GAN framework, CycleGAN to solve the challenging nonlinear unmixing problem. We exploit the linkage between the cycle consistency loss used in CycleGAN and the spectral reconstruction loss used in traditional methods. We make the essential discovery that the usage of the cycle consistency loss enables the learning of the mixing and unmixing processes to be dependent on the training data only, without the need of an explicit mixing model. We refer to the proposed approach as CycleGAN unmixing net, or CGU net. Experimental results indicate that the proposed CGU net exhibits stable and competitive performance on different datasets as compared to traditional HNU methods that are model-based.

Truth discovery is an effective tool to unearth truthful answers in crowdsourced question answering systems. Incentive mechanisms are necessary in such systems to stimulate worker participation. However, most of existing incentive mechanisms only consider compensating workers' resource cost, while the cost incurred by potential privacy leakage has been rarely incorporated. More importantly, to the best of our knowledge, how to provide personalized payments for workers with different privacy demands remains uninvestigated thus far. In this paper, we propose a contract-based personalized privacy-preserving incentive mechanism for truth discovery in crowdsourced question answering systems, named PINTION, which provides personalized payments for workers with different privacy demands as a compensation for privacy cost, while ensuring accurate truth discovery. The basic idea is that each worker chooses to sign a contract with the platform, which specifies a privacy-preserving level (PPL) and a payment, and then submits perturbed answers with that PPL in return for that payment. Specifically, we respectively design a set of optimal contracts under both complete and incomplete information models, which could maximize the truth discovery accuracy, while satisfying the budget feasibility, individual rationality and incentive compatibility properties. Experiments on both synthetic and real-world datasets validate the feasibility and effectiveness of PINTION.

Although deep learning-based approaches have been very effective in solving problems with plenty of labeled data, they suffer in tackling problems for which labeled data are scarce. In few-shot classification, the objective is to train a classifier from only a handful of labeled examples in a support set. In this paper, we propose a few-shot learning framework based on structured margin loss which takes into account the global structure of the support set in order to generate a highly discriminative feature space where the features from distinct classes are well separated in clusters. Moreover, in our meta-learning-based framework, we propose a context-aware query embedding encoder for incorporating support set context into query embedding and generating more discriminative and task-dependent query embeddings. The task-dependent features help the metalearner to learn a distribution over tasks more effectively. Extensive experiments based on few-shot, zero-shot and semi-supervised learning on three benchmarks show the advantages of the proposed model compared to state-of-the- art.

Federated learning has emerged as an advanced privacy-preserving learning technique for mobile edge computing, where the model is trained in a decentralized manner by the clients, preventing the server from directly accessing those private data from the clients. This learning mechanism significantly challenges the attack from the server side. Although the state-of-the-art attacking techniques that incorporated the advance of Generative adversarial networks (GANs) could construct class representatives of the global data distribution among all clients, it is still challenging to distinguishably attack a specific client (i.e., user-level privacy leakage), which is a stronger privacy threat to precisely recover the private data from a specific client. To analyze the privacy leakage of federated learning, this paper gives the first attempt to explore user-level privacy leakage by the attack from a malicious server. We propose a framework incorporating GAN with a multi-task discriminator, called multi-task GAN - Auxiliary Identification (mGAN-AI), which simultaneously discriminates category, reality, and client identity of input samples. The novel discrimination on client identity enables the generator to recover user specified private data. Unlike existing works interfering the federated learning process, the proposed method works “invisibly” on the server side. Furthermore, considering the anonymization strategy for mitigating mGAN-AI, we propose a beforehand linkability attack which re-identifies the anonymized updates by associating the client representatives. A novel siamese network fusing the identification and verification models is developed for measuring the similarity of representatives. The experimental results demonstrate the effectiveness of the proposed approaches and the superior to the state-of-the-art.

Due to the significant information loss in low-resolution (LR) images, it has become extremely challenging to further advance the state-of-the-art of single image super-resolution (SISR). Reference-based super-resolution (RefSR), on the other hand, has proven to be promising in recovering high-resolution (HR) details when a reference (Ref) image with similar content as that of the LR input is given. However, the quality of RefSR can degrade severely when Ref is less similar. This paper aims to unleash the potential of RefSR by leveraging more texture details from Ref images with stronger robustness even when irrelevant Ref images are provided. Inspired by the recent work on image stylization, we formulate the RefSR problem as neural texture transfer. We design an end-to-end deep model which enriches HR details by adaptively transferring the texture from Ref images according to their textural similarity. Instead of matching content in the raw pixel space as done by previous methods, our key contribution is a multi-level matching conducted in the neural space. This matching scheme facilitates multi-scale neural transfer that allows the model to benefit more from those semantically related Ref patches, and gracefully degrade to SISR performance on the least relevant Ref inputs. We build a benchmark dataset for the general research of RefSR, which contains Ref images paired with LR inputs with varying levels of similarity. Both quantitative and qualitative evaluations demonstrate the superiority of our method over state-of-the-art.

Person re-identification (re-ID) is the task of matching person images across camera views, which plays an important role in surveillance and security applications. Inspired by great progress of deep learning, deep re-ID models began to be popular and gained state-of-the-art performance. However, recent works found that deep neural networks (DNNs) are vulnerable to adversarial examples, posing potential threats to DNNs based applications. This phenomenon throws a serious question about whether deep re-ID based systems are vulnerable to adversarial attacks. In this paper, we take the first attempt to implement robust physical-world attacks against deep re-ID. We propose a novel attack algorithm, called advPattern, for generating adversarial patterns on clothes, which learns the variations of image pairs across cameras to pull closer the image features from the same camera, while pushing features from different cameras farther. By wearing our crafted “invisible cloak”, an adversary can evade person search, or impersonate a target person to fool deep re-ID models in physical world. We evaluate the effectiveness of our transformable patterns on adversaries' clothes with Market1501 and our established PRCS dataset. The experimental results show that the rank-1 accuracy of re-ID models for matching the adversary decreases from 87.9% to 27.1% under Evading Attack. Furthermore, the adversary can impersonate a target person with 47.1% rank-1 accuracy and 67.9% mAP under Impersonation Attack. The results demonstrate that deep re-ID systems are vulnerable to our physical attacks.

Land-cover classification to distinguish physical covers of Earth's surface is one of the critical tasks in remote sensing. Although deep learning-based approaches have shown remarkable performance in semantic segmentation, they require a massive amount of training data. Thus, the generalization capability of these approaches is of great importance, especially in working with satellite images when the amount of available labeled data is quite limited. In this paper, we propose incorporating spectral unmixing methods to obtain powerful representations of spectral information for semantic segmentation of satellite images. We show that land-cover classification performance can be enhanced by this proper extraction of features as input to the deep learning-based model. The experimental results demonstrate promising potential improvements in terms of segmentation accuracy. In addition, qualitative assessments show a higher confidence level of the proposed framework in predicting a label for a given pixel.

Region proposal algorithms play an important role in most state-of-the-art two-stage object detection networks by hypothesizing object locations in the image. Nonetheless, region proposal algorithms are known to be the bottleneck in most two-stage object detection networks, increasing the processing time for each image and resulting in slow networks not suitable for real-time applications such as autonomous driving vehicles. In this paper we introduce RRPN, a Radar-based real-time region proposal algorithm for object detection in autonomous driving vehicles. RRPN generates object proposals by mapping Radar detections to the image coordinate system and generating pre-defined anchor boxes for each mapped Radar detection point. These anchor boxes are then transformed and scaled based on the object's distance from the vehicle, to provide more accurate proposals for the detected objects. We evaluate our method on the newly released NuScenes dataset [1] using the Fast R-CNN object detection network [2]. Compared to the Selective Search object proposal algorithm [3], our model operates more than 100× faster while at the same time achieves higher detection precision and recall. Code has been made publicly available at https://github.com/mrnabati/RRPN.

Identifying land-cover and specifically the type of the material that constitutes building roofs in urban areas provides important reference information for later procedures including semantic labeling, bridge masking, and 3D reconstruction. In this paper, we present a hybrid unmixing-based classification framework that integrates both class-wise unsupervised unmixing and supervised unmixing that effectively convert the classification problem from the original spectral space to the abundance space, such that the intrinsic characteristics of each material can be better represented. Experimental results demonstrate competitive performance in terms of classification accuracy. In addition, we show that the proposed approach has the capability of handling new region of interest with similar scene content but different illumination geometry and atmospheric composition, which is crucial in classification of satellite images with a limited amount of training data.

Nonlinear unmixing of hyperspectral images has been a very challenging research problem, as it needs to consider the physical interactions between the sunlight scattered by multiple materials. In this paper, we propose a new approach for nonlinear unmixing which is based on multi-task learning (MTL) with low-rank matrix factorization (LRMF). The proposed approach establishes two tasks to conduct the unmixing problem under a nonlinear mixing model. In the first task, we employ LRMF to obtain endmember signatures and their corresponding abundance fractions simultaneously. Then, the second task uses LRMF to solve interactions from multiple scattering. The effectiveness of the proposed method is verified by using real hyperspectral data. Compared with other state-of-the-art nonlinear unmixing algorithms, the proposed approach demonstrates very competitive performance.

Given an arbitrary face image and an arbitrary speech clip, the proposed work attempts to generate the talking face video with accurate lip synchronization. Existing works either do not consider temporal dependency across video frames thus yielding abrupt facial and lip movement or are limited to the generation of talking face video for a specific person thus lacking generalization capacity. We propose a novel conditional recurrent generation network that incorporates both image and audio features in the recurrent unit for temporal dependency. To achieve both image- and video-realism, a pair of spatial-temporal discriminators are included in the network for better image/video quality. Since accurate lip synchronization is essential to the success of talking face video generation, we also construct a lip-reading discriminator to boost the accuracy of lip synchronization. We also extend the network to model the natural pose and expression of talking face on the Obama Dataset. Extensive experimental results demonstrate the superiority of our framework over the state-of-the-arts in terms of visual quality, lip sync accuracy, and smooth transition pertaining to both lip and facial movement.

Federated learning, i.e., a mobile edge computing framework for deep learning, is a recent advance in privacy-preserving machine learning, where the model is trained in a decentralized manner by the clients, i.e., data curators, preventing the server from directly accessing those private data from the clients. This learning mechanism significantly challenges the attack from the server side. Although the state-of-the-art attacking techniques that incorporated the advance of Generative adversarial networks (GANs) could construct class representatives of the global data distribution among all clients, it is still challenging to distinguishably attack a specific client (i.e., user-level privacy leakage), which is a stronger privacy threat to precisely recover the private data from a specific client. This paper gives the first attempt to explore user-level privacy leakage against the federated learning by the attack from a malicious server. We propose a framework incorporating GAN with a multi-task discriminator, which simultaneously discriminates category, reality, and client identity of input samples. The novel discrimination on client identity enables the generator to recover user specified private data. Unlike existing works that tend to interfere the training process of the federated learning, the proposed method works “invisibly” on the server side. The experimental results demonstrate the effectiveness of the proposed attacking approach and the superior to the state-of-the-art.

Road-users are a critical part of decision-making for both self-driving cars and driver assistance systems. Some road-users, however, are more important for decision-making than others because of their respective intentions, ego-vehicle's intention and their effects on each other. In this paper, we propose a novel architecture for road-user importance estimation which takes advantage of the local and global context of the scene. For local context, the model exploits the appearance of the road users (which captures orientation, intention, etc.) and their location relative to ego-vehicle. The global context in our model is defined based on the feature map of the convolutional layer of the module which predicts the future path of the ego-vehicle and contains rich global information of the scene (e.g., infrastructure, road lanes, etc.), as well as the ego-vehicle's intention information. Moreover, this paper introduces a new data set of real-world driving, concentrated around intersections and includes annotations of important road users. Systematic evaluations of our proposed method against several baselines show promising results.

Linear spectral unmixing is the practice of decomposing the mixed pixel into a linear combination of the constituent endmembers and the estimated abundances. This paper focuses on unsupervised spectral unmixing where the endmembers are unknown a priori. Conventional approaches use either geometrical- or statistical-based approaches. In this paper, we address the challenges of spectral unmixing with unsupervised deep learning models, in specific, the autoencoder models, where the decoder serves as the endmembers and the hidden layer output serves as the abundances. In several recent attempts, part-based autoencoders have been designed to solve the unsupervised spectral unmixing problem. However, the performance has not been satisfactory. In this paper, we first discuss some important findings we make on issues with part-based autoencoders. By proof of counterexample, we show that all existing part-based autoencoder networks with nonnegative and tied encoder and decoder are inherently defective by making these inappropriate assumptions on the network structure. As a result, they are not suitable for solving the spectral unmixing problem. We propose a so-called untied denoising autoencoder with sparsity, in which the encoder and decoder of the network are independent, and only the decoder of the network is enforced to be nonnegative. Furthermore, we make two critical additions to the network design. First, since denoising is an essential step for spectral unmixing, we propose to incorporate the denoising capacity into the network optimization in the format of a denoising constraint rather than cascading another denoising preprocessor in order to avoid the introduction of additional reconstruction error. Second, to be more robust to the inaccurate estimation of a number of endmembers, we adopt an $l_{21}$ -norm on the encoder of the network to reduce the redundant endmembers while decreasing the reconstruction error simultaneously. The experimental results demonstrate that the proposed approach outperforms several state-of-the-art methods, especially for highly noisy data.

Location information of workers are usually required for optimal task allocation in mobile crowdsensing, which however raises severe concerns of location privacy leakage. Although many approaches have been proposed to protect the locations of users, the location protection for task allocation in mobile crowdsensing has not been well explored. In addition, to the best of our knowledge, none of existing privacy-preserving task allocation mechanisms can provide personalized location protection considering different protection demands of workers. In this paper, we propose a personalized privacy-preserving task allocation framework for mobile crowdsensing that can allocate tasks effectively while providing personalized location privacy protection. The basic idea is that each worker uploads the obfuscated distances and personal privacy level to the server instead of its true locations or distances to tasks. In particular, we propose a Probabilistic Winner Selection Mechanism (PWSM) to minimize the total travel distance with the obfuscated information from workers, by allocating each task to the worker who has the largest probability of being closest to it. Moreover, we propose a Vickrey Payment Determination Mechanism (VPDM) to determine the appropriate payment to each winner by considering its movement cost and privacy level, which satisfies the truthfulness, profitability, and probabilistic individual rationality. Extensive experiments on the real-world datasets demonstrate the effectiveness of the proposed mechanisms.

The continuous publication of aggregate statistics over crowd-sourced data to the public has enabled many data mining applications (e.g., real-time traffic analysis). Existing systems usually rely on a trusted server to aggregate the spatio-temporal crowd-sourced data and then apply differential privacy mechanism to perturb the aggregate statistics before publishing to provide strong privacy guarantee. However, the privacy of users will be exposed once the server is hacked or cannot be trusted. In this paper, we study the problem of real-time crowd-sourced statistical data publishing with strong privacy protection under an untrusted server. We propose a novel distributed agent-based privacy-preserving framework, called DADP, that introduces a new level of multiple agents between the users and the untrusted server. Instead of directly uploading the check-in information to the untrusted server, a user can randomly select one agent and upload the check-in information to it with the anonymous connection technology. Each agent aggregates the received crowd-sourced data and perturbs the aggregated statistics locally with Laplace mechanism. The perturbed statistics from all the agents are further combined together to form the entire perturbed statistics for publication. In particular, we propose a distributed budget allocation mechanism and an agent-based dynamic grouping mechanism to realize global w-event ε-differential privacy in a distributed way. We prove that DADP can provide w-event ε-differential privacy for real-time crowd-sourced statistical data publishing under the untrusted server. Extensive experiments on real-world datasets demonstrate the effectiveness of DADP..

In many computer vision applications, obtaining images of high resolution in both the spatial and spectral domains are equally important. However, due to hardware limitations, one can only expect to acquire images of high resolution in either the spatial or spectral domains. This paper focuses on hyperspectral image super-resolution (HSI-SR), where a hyperspectral image (HSI) with low spatial resolution (LR) but high spectral resolution is fused with a multispectral image (MSI) with high spatial resolution (HR) but low spectral resolution to obtain HR HSI. Existing deep learning-based solutions are all supervised that would need a large training set and the availability of HR HSI, which is unrealistic. Here, we make the first attempt to solving the HSI-SR problem using an unsupervised encoder-decoder architecture that carries the following uniquenesses. First, it is composed of two encoder-decoder networks, coupled through a shared decoder, in order to preserve the rich spectral information from the HSI network. Second, the network encourages the representations from both modalities to follow a sparse Dirichlet distribution which naturally incorporates the two physical constraints of HSI and MSI. Third, the angular difference between representations are minimized in order to reduce the spectral distortion. We refer to the proposed architecture as unsupervised Sparse Dirichlet-Net, or uSDN. Extensive experimental results demonstrate the superior performance of uSDN as compared to the state-of-the-art.

Building on top of the success of generative adversarial networks (GANs), conditional GANs attempt to better direct the data generation process by conditioning with certain additional information. Inspired by the most recent AC-GAN, in this paper we propose a fast-converging conditional GAN (FC-GAN). In addition to the real/fake classifier used in vanilla GANs, our discriminator has an advanced auxiliary classifier which distinguishes each real class from an extra `fake' class. The `fake' class avoids mixing generated data with real data, which can potentially confuse the classification of real data as AC-GAN does, and makes the advanced auxiliary classifier behave as another real/fake classifier. As a result, FC-GAN can accelerate the process of differentiation of all classes, thus boost the convergence speed. Experimental results on image synthesis demonstrate our model is competitive in the quality of images generated while achieving a faster convergence rate.

Learning compact representation is vital and challenging for large scale multimedia data. Cross-view/crossmodal hashing for effective binary representation learning has received significant attention with exponentially growing availability of multimedia content. Most existing crossview hashing algorithms emphasize the similarities in individual views, which are then connected via cross-view similarities. In this work, we focus on the exploitation of the discriminative information from different views, and propose an end-to-end method to learn semantic-preserving and discriminative binary representation, dubbed Discriminative Cross-View Hashing (DCVH), in light of learning multitasking binary representation for various tasks including cross-view retrieval, image-to-image retrieval, and image annotation/tagging. The proposed DCVH has the following key components. First, it uses convolutional neural network (CNN) based nonlinear hashing functions and multilabel classification for both images and texts simultaneously. Such hashing functions achieve effective continuous relaxation during training without explicit quantization loss by using Direct Binary Embedding (DBE) layers. Second, we propose an effective view alignment via Hamming distance minimization, which is efficiently accomplished by bit-wise XOR operation. Extensive experiments on two image-text benchmark datasets demonstrate that DCVH outperforms state-of-the-art cross-view hashing algorithms as well as single-view image hashing algorithms. In addition, DCVH can provide competitive performance for image annotation/tagging.

Incorporating encoding-decoding nets with adversarial nets has been widely adopted in image generation tasks. We observe that the state-of-the-art achievements were obtained by carefully balancing the reconstruction loss and adversarial loss, and such balance shifts with different network structures, datasets, and training strategies. Empirical studies have demonstrated that an inappropriate weight between the two losses may cause instability, and it is tricky to search for the optimal setting, especially when lacking prior knowledge on the data and network. This paper gives the first attempt to relax the need of manual balancing by proposing the concept of decoupled learning, where a novel network structure is designed that explicitly disentangles the backpropagation paths of the two losses. In existing works, the encoding-decoding nets and GANs are integrated by sharing weights on the generator/decoder, thus the two losses are backpropagated to the generator/decoder simultaneously, where a weighting factor is needed to balance the interaction between the two losses. The decoupled learning avoids the interaction and thus removes the requirement of the weighting factor, essentially improving the generalization capacity of the designed model to different applications. The decoupled learning framework could be easily adapted to most existing encoding-decoding-based generative networks and achieve competitive performance without the need of weight adjustment. Experimental results demonstrate the effectiveness, robustness, and generality of the proposed method. The other contribution of the paper is the design of a new evaluation metric to measure the image quality of generative models. We propose the so-called normalized relative discriminative score (NRDS), which introduces the idea of relative comparison, rather than providing absolute estimates like existing metrics.

We develop an aerial image super-resolution method by training convolutional neural networks (CNNs) with respect to wavelet analysis. To this end, we commence by performing wavelet decomposition to aerial images for multiscale representations. We then train multiple CNNs for approximating the wavelet multiscale representations, separately. The multiple CNNs thus trained characterize aerial images in multiple directions and multiscale frequency bands, and thus enable image restoration subject to sophisticated culture variability. For inference, the trained CNNs regress wavelet multiscale representations from a low-resolution aerial image, followed by wavelet synthesis that forms a restored high-resolution aerial image. Experimental results validate the effectiveness of our method for restoring complicated aerial images.

Anomaly detection has been known to be a challenging problem due to the uncertainty of anomaly and the interference of noise. In this paper, we focus on anomaly detection in hyperspectral images (HSIs) and propose a novel detection algorithm based on spectral unmixing and dictionary-based low-rank decomposition. The innovation is threefold. First, due to the highly mixed nature of pixels in HSI data, instead of using the raw pixel directly for anomaly detection, the proposed algorithm applies spectral unmixing to obtain the abundance vectors and uses these vectors for anomaly detection. We show that the abundance vectors possess more distinctive features to identify anomaly from background. Second, to better represent the highly correlated background and the sparse anomaly, we construct a dictionary based on the mean shift clustering of the abundance vectors to improve both the discriminative and representative powers of the algorithm. Finally, a low-rank matrix decomposition method based on the constructed dictionary is proposed to encourage the coefficients of the dictionary, instead of the background itself, to be low rank, and the residual matrix to be sparse. Anomalies can then be extracted by summing up the columns of the residual matrix. The proposed algorithm is evaluated on both synthetic and real data sets. Experimental results show that the proposed approach constantly achieves high detection rate, while maintaining low false alarm rate regardless of the type of images tested.

We develop an automatic oil spill segmentation method in terms of f-divergence minimization. We exploit f-divergence for measuring the disagreement between the distributions of ground-truth and generated oil spill segmentations. To render tractable optimization, we minimize the tight lower bound of the f-divergence by adversarial training a regressor and a generator, which are structured in different forms of deep neural networks separately. The generator aims at producing accurate oil spill segmentation, while the regressor characterizes discriminative distributions with respect to true and generated oil spill segmentations. It is the coplay between the generator net and the regressor net against each other that achieves a minimal of the maximum lower bound for the f-divergence. The adversarial strategy enhances the representational powers of both the generator and the regressor and avoids requesting large amounts of labeled data for training the deep network parameters. In addition, the trained generator net enables automatic oil spill detection that does not require manual initialization. Benefiting from the comprehensiveness of f-divergence for characterizing diversified distributions, our framework can accurately segment variously shaped oil spills in noisy synthetic aperture radar images. Experimental results validate the effectiveness of the proposed oil spill segmentation framework.

If I provide you a face image of mine (without telling you the actual age when I took the picture) and a large amount of face images that I crawled (containing labeled faces of different ages but not necessarily paired), can you show me what I would look like when I am 80 or what I was like when I was 5? The answer is probably a No. Most existing face aging works attempt to learn the transformation between age groups and thus would require the paired samples as well as the labeled query image. In this paper, we look at the problem from a generative modeling perspective such that no paired samples is required. In addition, given an unlabeled image, the generative model can directly produce the image with desired age attribute. We propose a conditional adversarial autoencoder (CAAE) that learns a face manifold, traversing on which smooth age progression and regression can be realized simultaneously. In CAAE, the face is first mapped to a latent vector through a convolutional encoder, and then the vector is projected to the face manifold conditional on age through a deconvolutional generator. The latent vector preserves personalized face features (i.e., personality) and the age condition controls progression vs. regression. Two adversarial networks are imposed on the encoder and generator, respectively, forcing to generate more photo-realistic faces. Experimental results demonstrate the appealing performance and flexibility of the proposed framework by comparing with the state-of-the-art and ground truth.

Distributed surveillance systems have become popular in recent years due to security concerns. However, transmitting high dimensional data in bandwidth-limited distributed systems becomes a major challenge. In this paper, we address this issue by proposing a novel probabilistic algorithm based on the divergence between the probability distributions of the visual features in order to reduce their dimensionality and thus save the network bandwidth in distributed wireless smart camera networks. We demonstrate the effectiveness of the proposed approach through extensive experiments on two surveillance recognition tasks.

Learning binary representation is essential to large-scale computer vision tasks. Most existing algorithms require a separate quantization constraint to learn effective hashing functions. In this work, we present Direct Binary Embedding (DBE), a simple yet very effective algorithm to learn binary representation in an end-to-end fashion. By appending an ingeniously designed DBE layer to the deep convolutional neural network (DCNN), DBE learns binary code directly from the continuous DBE layer activation without quantization error. By employing the deep residual network (ResNet) as DCNN component, DBE captures rich semantics from images. Furthermore, in the effort of handling multilabel images, we design a joint cross entropy loss that includes both softmax cross entropy and weighted binary cross entropy in consideration of the correlation and independence of labels, respectively. Extensive experiments demonstrate the significant superiority of DBE over state-of-the-art methods on tasks of natural object recognition, image retrieval and image annotation.

Despite recent attempts for solving the person re-identification problem, it remains a challenging task since a person's appearance can vary significantly when large variations in view angle, human pose and illumination are involved. The concept of attention is one of the most interesting recent architectural innovations in neural networks. Inspired by that, in this paper we propose a novel approach based on using a gradient-based attention mechanism in deep convolution neural network for solving the person re-identification problem. Our model learns to focus selectively on parts of the input image for which the networks' output is most sensitive to. Extensive comparative evaluations demonstrate that the proposed method outperforms state-of-the-art approaches, including both traditional and deep neural network-based methods on the challenging CUHK01 and CUHK03 datasets.

This paper presents a novel hierarchical approach for the simultaneous tracking of multiple targets in a video. We use a network flow approach to link detections in low-level and tracklets in high-level. At each step of the hierarchy, the confidence of candidates is measured by using a new scoring system, ConfRank, that considers the quality and the quantity of its neighborhood. The output of the first stage is a collection of safe tracklets and unlinked high-confidence detections. For each individual detection, we determine if it belongs to an existing or is a new tracklet. We show the effect of our framework to recover missed detections and reduce switch identity. The proposed tracker is referred to as TVOD for multi-target tracking using the visual tracker and generic object detector. We achieve competitive results with lower identity switches on several datasets comparing to state-of-the-art.

Nowadays, distributed smart cameras are deployed for a wide set of tasks in several application scenarios, ranging from object recognition, image retrieval, and forensic applications. Due to limited bandwidth in distributed systems, efficient coding of local visual features has in fact been an active topic of research. In this paper, we propose a novel approach to obtain a compact representation of high-dimensional visual data using sensor fusion techniques. We convert the problem of visual analysis in resource-limited scenarios to a multiview representation learning, and we show that the key to finding properly compressed representation is to exploit the position of cameras with respect to each other as a norm-based regularization in the particular signal representation of sparse coding. Learning the representation of each camera is viewed as an individual task and a multi-task learning with joint sparsity for all nodes is employed. The proposed representation learning scheme is referred to as the multi-view task-driven learning for visual sensor network (MT-VSN). We demonstrate that MT-VSN outperforms state-of-the-art in various surveillance recognition tasks.

Spectral unmixing is to decompose the hyperspectral data into endmembers and abundances. It has been known to be a challenging and ill-posed task due to the corruption of noise as well as complex environmental conditions. In this paper, we propose a part-based denoising autoencoder with unique structure that solves the unmixing challenges. The effective l21norm and denoising constraints are applied on the network to better handle noise, while at the same time reducing the reconstruction error and redundant endmembers simultaneously. A back propagation optimization method powered with the Armijo rule is proposed to project the weights to the non-negativity space that guarantees the sum-to-one constraint. The experimental results demonstrate the proposed approach is able to outperform several state-of-the-art methods for highly noisy data.

Pansharpening refers to the fusion of a high spatial resolution panchromatic image with high spectral resolution multispectral or hyperspectral images (MSI or HSI) to yield high resolution data in both spectral and spatial domains. It has been widely adopted as a primary preprocessing step for numerous applications. In this paper, we perform a literature survey of various pansharpening algorithms including the most advanced deep learning approaches for both multispectral and hyperspectral images. We further evaluate the effect of the resolution difference on anomaly detection. Synthetic multispectral and hyperspectral images are generated to evaluate the performance of anomaly detection on high resolution images. Eight state-of-the-art MSI and HSI pansharpening methods are compared in this paper. Experimental results show that, performing anomaly detection on high resolution images improves the detection rate, and at the mean time suppresses the false alarm rate.

Binary descriptors not only are beneficial for similarity search, they are also capable of serving as discriminant features for classification purpose. In this paper we propose a new algorithm based on cross entropy to learn effective binary descriptors, dubbed CE-Bits, providing an alternative to L-2 and hinge loss learning. Because of the usage of cross entropy, a min-max binary NP-hard problem is raised to optimize the binary code during training. We provide a novel solution by breaking the binary code into independent blocks and optimize them individually. Although sub-optimal, our method converges very fast and outperforms its L-2 and hinge loss counterparts. By conducting extensive experiments on several benchmark datasets, we show that CE-Bits efficiently generates effective binary descriptors for both classification and retrieval tasks and outper-forms state-of-the-art supervised hashing algorithms.

As one of the most popular cloud services, data storage has attracted great attention in recent research efforts. Key-value (k-v) stores have emerged as a popular option for storing and querying billions of key-value pairs. So far, existing methods have been deterministic. Providing such accuracy, however, comes at the cost of memory and CPU time. In contrast, we present an approximate k-v storage for cloud-based systems that is more compact than existing methods. The tradeoff is that it may, theoretically, return errors. Its design is based on the probabilistic data structure called “bloom filter”, where we extend the classical bloom filter to support key-value operations. We call the resulting design as the kBF (key-value bloom filter). We further develop a distributed version of the kBF (d-kBF) for the unique requirements of cloud computing platforms, where multiple servers cooperate to handle a large volume of queries in a load-balancing manner. Finally, we apply the kBF to a practical problem of implementing a state machine to demonstrate how the kBF can be used as a building block for more complicated software infrastructures.

Classical dictionary learning algorithms that rely on a single source of information have been successfully used for the discriminative tasks. However, exploiting multiple sources has demonstrated its effectiveness in solving challenging real-world situations. We propose a new framework for feature fusion to achieve better classification performance as compared to the case where individual sources are utilized. In the context of multimodal data analysis, the modality configuration induces a strong group/coupling structure. The proposed method models the coupling between different modalities in space of sparse codes while at the same time within each modality a discriminative dictionary is learned in an all-vs-all scheme whose class-specific sub-parts are non-correlated. The proposed dictionary learning scheme is referred to as the multimodal weighted dictionary learning (MWDL). We demonstrate that MWDL outperforms state-of-the-art dictionary learning approaches in various experiments.

Drift is the most difficult issue in object visual tracking based on framework of “tracking-by-detection”. Due to the self-taught learning, the mis-aligned samples are potentially to be incorporated in learning and degrade the discrimination of the tracker. This paper proposes a new tracking approach that resolves this problem by three multi-level collaborative components: a high-level global appearance tracker provides a basic prediction, upon which the structure preserved low-level local patches matching helps to guarantee precise tracking with minimized drift. Those local patches are deliberately deployed on the foreground object via foreground/background segmentation, which is realized by a simple and efficient classifier trained by super-pixel segments. Experimental results show that the three closely collaborated components enable our tracker runs in real time and performs favourably against state-of-the-art approaches on challenging benchmark sequences.

The staggering amount of streaming time series coming from the real world calls for more efficient and effective online modeling solution. For time series modeling, most existing works make some unrealistic assumptions such as the input data is of fixed length or well aligned, which requires extra effort on segmentation or normalization of the raw streaming data. Although some literature claim their approaches to be invariant to data length and misalignment, they are too time-consuming to model a streaming time series in an online manner. We propose a novel and more practical online modeling and classification scheme, DDE-MGM, which does not make any assumptions on the time series while maintaining high efficiency and state-of-the-art performance. The derivative delay embedding (DDE) is developed to incrementally transform time series to the embedding space, where the intrinsic characteristics of data is preserved as recursive patterns regardless of the stream length and misalignment. Then, a non-parametric Markov geographic model (MGM) is proposed to both model and classify the pattern in an online manner. Experimental results demonstrate the effectiveness and superior classification accuracy of the proposed DDE-MGM in an online setting as compared to the state-of-the-art.

Distributed object recognition is a significantly fast-growing research area, mainly motivated by the emergence of high performance cameras and their integration with modern wireless sensor network technologies. In wireless distributed object recognition, the bandwidth is limited and it is desirable to avoid transmitting redundant visual features from multiple cameras to the base station. In this paper, we propose a histogram compression and feature selection framework based on Sparse Non-negative Matrix Factorization (SNMF). In our proposed method, histograms of the features are modeled as linear combination of a small set of signature vectors with associated weight vectors. The recognition process in the base station is then performed based on these small sets of transmitted weights from each camera. Furthermore, we propose another novel distributed object recognition scheme based on local classification in each camera and sending the label information to the base station and making the final decision based on majority voting. Experiments on BMW dataset affirm that our approach outperforms the state of the art in accuracy and bandwidth usage.

Classical dictionary learning algorithms that rely on a single source of information have been successfully used for classification tasks. Additionally, the exploitation of multiple sources has shown to be advantageous in challenging real-world situations. We propose a new framework to exploit robust modality fusion in classification in order to achieve better classification performance than single source methods. Multimodal learning is able to leverage any correlations between sensor modalities found in the data. We propose a new bilevel optimization, referred to as (MCJWDL). We perform supervised dictionary learning while forcing a coupling between the resulting sparse codes from different sources of information. Extensive experiments demonstrate that MCJWDL outperforms state-of-the-art sparse representation and dictionary learning approaches for the multi-view object and multi-view action recognition.

Anomaly detection has been known to be a challenging, ill-posed problem due to the uncertainty of anomaly and the interference of noise. In this paper, we propose a novel low rank anomaly detection algorithm in hyperspectral images (HSI), where three components are involved. First, due to the highly mixed nature of pixels in HSI, instead of using the raw pixel directly for anomaly detection, the proposed algorithm applies spectral unmixing algorithms to obtain the abundance vectors and uses these vectors for anomaly detection. Second, for better classification, a dictionary is built based on the mean-shift clustering of the abundance vectors to better represent the highly-correlated background and the sparse anomaly. Finally, a low-rank matrix decomposition is proposed to encourage the sparse coefficients of the dictionary to be low-rank, and the residual matrix to be sparse. Anomalies can then be extracted by summing up the columns of the residual matrix. The proposed algorithm is evaluated on both synthetic and real datasets. Experimental results show that the proposed approach constantly achieves high detection rate while maintaining low false alarm rate regardless of the type of images tested.

Automatic facial image analysis has received considerable research interests due to its important role in computer vision and biometrics. As the key component, face feature is usually extracted under largely controlled environment and learnt for specific tasks which limits its discriminant capability in a multi-task learning scenario. In this paper, we present a novel deeply learnt tree-structured face representation to model the human face with multiple semantic meanings, such as identity, expression and age, that wouldyield a unified feature representation of the facial image. The tree structure is built based on the incorporation of an unsupervised shallow network that generates the low-level features serving as the leaf nodes and the recursive application of the designed semi-supervised AutoEncoder to generate the intermediate and root nodes. By incorporating the label information with different semantic meanings, the designed semi-supervised AutoEncoder aims to distinguish the latent factors embedded in facial images with automatically learned tree structure and weights. To validate the effectiveness of the proposed facial representation, we design comprehensive experiments based on the FACES dataset which is considered as the most challenging benchmark that reflects multiple biometric factors. We show that the proposed feature yields unified representation in multitask facial biometrics. The multi-task learning framework is applicable to many other computer vision tasks.

In this paper, we propose a new approach for the person re-identification problem, discovering the correct matches for a query pedestrian image from a set of gallery images. It is well motivated by our observation that the overall complex inter-camera transformation, caused by the change of camera viewpoints, person poses and view illuminations, can be effectively modelled by a combination of many simple local transforms, which guides us to learn a set of more specific local metrics other than a fixed metric working on the feature vector of a whole image. Given training images in pair, we first align the local patches using spatially constrained dense matching. Then, we use a decision tree structure to partition the space of the aligned local patch-pairs into different configurations according to the similarity of the local cross-view transforms. Finally, a local metric kernel is learned for each configuration at the tree leaf nodes in a linear regression manner. The pairwise distance between a query image and a gallery image is summarized based on all the pairwise distance of local patches measured by different local metric kernels. Multiple decision trees form the proposed random kernel forest, which always discriminatively assign the optimal local metric kernel to the local image patches in re-identification. Experimental results over the public benchmarks demonstrate the effectiveness of our approach for achieving very competitive performances with a relatively simpler learning scheme.

Robust visual tracking is a challenging problem due to pose variance, occlusion and cluttered backgrounds. No single feature can be robust to all possible scenarios in a video sequence. However, exploiting multiple features has demonstrated its effectiveness in overcoming challenging situations in visual tracking. We propose a new framework for multi-modal fusion at both the feature level and decision level by training a reconstructive and discriminative dictionary and classifier for each modality simultaneously with the additional constraint of label consistency across different modalities. In addition, a joint decision measure is designed based on both reconstruction and classification error to adaptively adjust the weights of different features such that unreliable features can be removed from tracking. The proposed tracking scheme is referred to as the label-consistent and fusion-based joint sparse coding (LC-FJSC). Extensive experiments on publicly available videos demonstrate that LC-FJSC outperforms state-of-the-art trackers.

Facial feature parsing is an active research topic in image understanding. In this paper, we address the feature parsing problem by applying low-rank matrix decomposition on facial images. Specifically, the face is considered as a combination of the skin background which resides in a low dimensional subspace with the salient feature components (e.g., eyes, nose and mouth) as the sparse noise. Given a face image, the feature parsing problem can then be naturally formulated as sparse noise detection when recovering a low-rank matrix. To enhance the feature parsing, a linear transformation matrix is learned to boost the discriminant feature extraction. Furthermore, with the derived parsing maps, the algorithm is easily extended to implement facial landmark detection task. The effectiveness of the proposed algorithm is evaluated through several experiments on comprehensive datasets.

Anomaly detection becomes increasingly important in hyper-spectral image analysis, since it can now uncover many material substances which were previously unresolved by multi-spectral sensors. In this paper, we propose a Low-rank Tensor Decomposition based anomaly Detection (LTDD) algorithm for Hyperspectral Imagery. The HSI data cube is first modeled as a dense low-rank tensor plus a sparse tensor. Based on the obtained low-rank tensor, LTDD further decomposes the low-rank tensor using Tucker decomposition to extract the core tensor which is treated as the “support” of the anomaly spectral signatures. LTDD then adopts an unmixing approach to the reconstructed core tensor for anomaly detection. The experiments based on both simulated and real hyperspectral data sets verify the effectiveness of our algorithm.

This paper presents a novel approach to simultaneous tracking of multiple targets in a video. Instead of using the unreliable “detector confidence scores,” it develops a new scoring system, ConfRank, that originates from the PageRank idea where not only the detection confidence score, but that the quality and the quantity of adjacent detections in spatio-temporal neighborhood are considered. The new scoring system effectively separates False Positives from True Positives, that enables us to remove unwanted detections using a simple threshold without loosing targets. Our framework outperforms state-of-the-art tracking methods in most evaluations. Specifically, it significantly reduces False Positives and switch identities while keeping missed detections low leading to higher precision and multiple object tracking accuracy (MOTA) on several standard datasets.

In many network applications, accurate traffic measurement is critical for bandwidth management with QoS requirements, and detecting security threats such as DoS (Denial of Service) attacks. In such cases, traffic is usually modeled as a collection of flows, which are identified based on certain features such as IP address pairs. One central problem is to identify those "heavy hitter" flows, which account for a large percentage of total traffic, e.g., at least 0.1% of the link capacity. However, the challenge for this goal is that keeping an individual counter for each flow is too slow, costly, and non-scalable. In this paper, we describe a novel data structure called the Probabilistic Bloom Filter (PBF), which extends the classical bloom filter into the probabilistic direction, so that it can effectively identify heavy hitters. We analyze the performance, tradeoffs, and capacity of this data structure. Our study also investigates how to calibrate this data structure's parameters. We also develop two extensions of the basic form of the PBF for more flexible application needs. We use real network traces collected on a Web query server and a backbone router to test the performance of the PBF, and demonstrate that this method can accurately keep track of all objects' frequencies, including websites and flows, so that heavy hitters can be identified with constant time computational complexity and low memory overhead.

This paper presents an approach for real-time multivehicle tracking and counting under fisheye camera based on simple feature points tracking, grouping and association. Different from traditional cameras, the main challenge under fisheye cameras is that the objects being tracked suffer from severe distortion and perspective effects in even adjacent frames. As a result, the points can be stably matched by a point tracker are much fewer, and the points even lose tracking completely quite occasionally. Firstly, to preserve points discrimination in dynamic grouping, we propose an approach based on motion similarity and neighbor weighted grafting to transfers motion knowledge between long and short point trajectories. Moreover, to deal with cases such as points losing tracking completely or incorrect points grouping, we also propose a concept of points "identity-appearance" that integrates constrained motion for association between vehicle track lets and segmented point groups. Our approach also overcomes several common challenges in traffic surveillance such as stopping vehicles, pedestrians and counting of linked (partially occluded) vehicles. Finally, extensive experimental results are provided on challenging fisheye image sequences to demonstrate the robustness and effectiveness of the approach.

In this paper, we study the compressed sensing (CS) image recovery problem. The traditional method divides the image into blocks and treats each block as an independent sub-CS recovery task. This often results in losing global structure of an image. In order to improve the CS recovery result, we propose a nonlocal (NL) estimation step after the initial CS recovery for denoising purpose. The NL estimation is based on the well-known NL means filtering that takes an advantage of self-similarity in images. We formulate the NL estimation as the low-rank matrix approximation problem, where the low-rank matrix is formed by the NL similarity patches. An efficient algorithm, nonlocal Douglas-Rachford (NLDR), based on Douglas-Rachford splitting is developed to solve this low-rank optimization problem constrained by the CS measurements. Experimental results demonstrate that the proposed NLDR algorithm achieves significant performance improvements over the state-of-the-art in CS image recovery.

Existing social networking services recommend friends to users based on their social graphs, which may not be the most appropriate to reflect a user's preferences on friend selection in real life. In this paper, we present Friendbook, a novel semantic-based friend recommendation system for social networks, which recommends friends to users based on their life styles instead of social graphs. By taking advantage of sensor-rich smartphones, Friendbook discovers life styles of users from user-centric sensor data, measures the similarity of life styles between users, and recommends friends to users if their life styles have high similarity. Inspired by text mining, we model a user's daily life as life documents, from which his/her life styles are extracted by using the Latent Dirichlet Allocation algorithm. We further propose a similarity metric to measure the similarity of life styles between users, and calculate users' impact in terms of life styles with a friend-matching graph. Upon receiving a request, Friendbook returns a list of people with highest recommendation scores to the query user. Finally, Friendbook integrates a feedback mechanism to further improve the recommendation accuracy. We have implemented Friendbook on the Android-based smartphones, and evaluated its performance on both small-scale experiments and large-scale simulations. The results show that the recommendations accurately reflect the preferences of users in choosing friends.

With advances in vehicle-to-vehicle communication, future vehicles will have access to a communication channel through which messages can be sent and received when two get close to each other. This enabling technology makes it possible for authenticated users to send queries to those vehicles of interest, such as those that are located within a geographic region, over multiple hops for various application goals. However, a naive method that requires flooding the queries to each active vehicle in a region will incur a total communication overhead that is proportional to the size of the area and the density of vehicles. In this paper, we study the problem of spatial queries for vehicle networks by investigating probabilistic methods, where we only try to obtain approximate estimates within desired confidence intervals using only sublinear overheads. We consider this to be particularly useful when spatial query results can be made approximate or not precise, as is the case with many potential applications. The proposed method has been tested on snapshots from real world vehicle network traces.

Barrier coverage is a critical issue in wireless sensor networks for security applications (e.g., border protection), the performance of which is highly related with locations of sensor nodes. Existing work on barrier coverage mainly assume that sensor nodes have accurate location information, however, little work explores the effects of location errors on barrier coverage. In this paper, we study the barrier coverage problem when sensor nodes have location errors and deploy mobile sensor nodes to improve barrier coverage if the network is not barrier covered after initial deployment. We analyze the relationship between the true distance and the measured distance of two stationary sensor nodes and derive the minimum number of mobile sensor nodes needed to connect them with a guarantee when nodes location errors. Furthermore, we propose a fault tolerant weighted barrier graph, based on which we prove that the minimum number of mobile sensor nodes needed to form barrier coverage with a guarantee is the length of the shortest path on the graph. Simulation results validate the correctness of our analysis.

In this demo, we present MoodMagician, a pervasive and unobtrusive mobile phone system for inferring human emotions through the recording, processing, and analysis of the real-time streaming Galvanic Skin Response (GSR) signal from human bodies. Being different from traditional multimodal emotion sensing systems which rely on data from multiple sensing sources and may hence interfere with people's daily life, our proposed system is able to detect various categories of human emotions using single GSR signal, which is captured by compact and wearable mobile sensing devices in an unobtrusive fashion. The proposed system has been evaluated by well-designed practical experiments to recognize human emotions. The recognition accuracy of each emotion can be up to 70% through the development of effective preprocessing algorithms and the extraction of representative features from the GSR signals.

Barrier coverage is a critical issue in wireless sensor networks for security applications (e.g., border protection) where directional sensors (e.g., cameras) are becoming more popular than omni-directional scalar sensors (e.g., microphones). However, barrier coverage cannot be guaranteed after initial random deployment of sensors, especially for directional sensors with limited sensing angles. In this paper, we study how to efficiently use mobile sensors to achieve \(k\) -barrier coverage. In particular, two problems are studied under two scenarios. First, when only the stationary sensors have been deployed, what is the minimum number of mobile sensors required to form \(k\) -barrier coverage? Second, when both the stationary and mobile sensors have been pre-deployed, what is the maximum number of barriers that could be formed? To solve these problems, we introduce a novel concept of weighted barrier graph (WBG) and prove that determining the minimum number of mobile sensors required to form \(k\) -barrier coverage is related with finding \(k\) vertex-disjoint paths with the minimum total length on the WBG. With this observation, we propose an optimal solution and a greedy solution for each of the two problems. Both analytical and experimental studies demonstrate the effectiveness of the proposed algorithms.

Collaboration in visual sensor networks is essential not only to compensate for the limitations of each sensor node but also to tolerate inaccurate information generated by faulty sensors. This article focuses on the design of a collaborative target localization algorithm that is resilient to sensor faults. We first develop a distributed solution to fault-tolerant target localization based on a so-called certainty map . To tolerate potential sensor faults, a voting mechanism is adopted and a threshold value needs to be specified which is the key to the realization of the distributed solution. Analytical study is conducted to derive the lower and upper bounds for the threshold such that the probability of faulty sensors negatively impacts the localization performance is less than a small value. Second, we focus on the detection and correction of one type of sensor faults, error in camera orientation. We construct a generative image model in each camera based on the detected target location to estimate camera's orientation, detect inaccuracies in camera orientations and correct them before they cascade. Based on results obtained from both simulation and real experiments, we show that the proposed method is effective in localization accuracy as well as fault detection and correction performance.

This paper addresses the problem of learning over-complete dictionaries for the coupled feature spaces, where the learned dictionaries also reflect the relationship between the two spaces. A Bayesian method using a beta process prior is applied to learn the over-complete dictionaries. Compared to previous couple feature spaces dictionary learning algorithms, our algorithm not only provides dictionaries that customized to each feature space, but also adds more consistent and accurate mapping between the two feature spaces. This is due to the unique property of the beta process model that the sparse representation can be decomposed to values and dictionary atom indicators. The proposed algorithm is able to learn sparse representations that correspond to the same dictionary atoms with the same sparsity but different values in coupled feature spaces, thus bringing consistent and accurate mapping between coupled feature spaces. Another advantage of the proposed method is that the number of dictionary atoms and their relative importance may be inferred non-parametrically. We compare the proposed approach to several state-of-the-art dictionary learning methods by applying this method to single image super-resolution. The experimental results show that dictionaries learned by our method produces the best super-resolution results compared to other state-of-the-art methods.

We consider the approximate sparse recovery problem in multi-hop Wireless Sensor Networks (WSNs) using Compressed Sensing/Compressive Sampling (CS). The goal is to recover the n-dimensional data values by querying only m ≪ n sensors based on some linear projection of sensor readings. To solve this problem, a distributed compressive sparse sampling (DCSS) algorithm is proposed based on sparse binary CS measurement matrix. Each sensor first samples the environment independently, then the fusion center (FC), acting as a pseudo-sensor, samples the sensor network to select a subset of sensors (m out of n) that respond to the FC through shortest path for data recovery purpose. The sparse binary matrix is designed using the unbalanced expander graph which achieves the state-of-the-art performance for CS schemes. This binary matrix can be interpreted as a sensor selection matrix whose fairness is analyzed. Extensive experiments on both synthetic and real data sets show that by querying only the minimum amount of m sensors using the DCSS algorithm, the CS recovery accuracy outperforms existing sparse random matrices and can be as good as those using random dense measurement matrices but using much less number of sensors. We also show that the sparse binary measurement matrix works well on compressible data which has the closest recovery result to the known best k-term approximation. The recovery is robust against noisy measurements and does not require regular WSN deployments (e.g., grids). The sparsity and binary properties of the measurement matrix contribute, to a great extent, the reduction of the in-network communication cost as well as the computational burden.

Human action recognition based on the depth information provided by commodity depth sensors is an important yet challenging task. The noisy depth maps, different lengths of action sequences, and free styles in performing actions, may cause large intra-class variations. In this paper, a new framework based on sparse coding and temporal pyramid matching (TPM) is proposed for depth-based human action recognition. Especially, a discriminative class-specific dictionary learning algorithm is proposed for sparse coding. By adding the group sparsity and geometry constraints, features can be well reconstructed by the sub-dictionary belonging to the same class, and the geometry relationships among features are also kept in the calculated coefficients. The proposed approach is evaluated on two benchmark datasets captured by depth cameras. Experimental results show that the proposed algorithm repeatedly achieves superior performance to the state of the art algorithms. Moreover, the proposed dictionary learning method also outperforms classic dictionary learning approaches.

Restricted Boltzmann Machine (RBM) has been successfully applied to unsupervised learning and intensity modeling of images. In this paper, we cast background subtraction as an image recovery and foreground residual estimation problem within the RBM hierarchy. We propose a partially-sparse RBM (PS-RBM) framework which models the image as the integration of the trained RBM weights where the weights are learnt from partially sparse and controlled redundancy network structure. The PS-RBM helps provide accurate background modeling even in dynamic and noisy environments. Experiments also validate the effectiveness of the proposed method on a comprehensive benchmark database.

Barrier coverage is a critical issue in wireless sensor networks for security applications (e.g., border protection) where directional sensors (e.g., cameras) are becoming more popular and advantageous than omni-directional scalar sensors for the extra dimensional information they provide. However, barrier coverage can not be guaranteed after initial random deployment of sensors, especially for directional sensors with limited sensing angles. In this paper, we study how to efficiently achieve barrier coverage in hybrid directional sensor networks by moving mobile sensors to fill in gaps and form a barrier with stationary sensors. In specific, we introduce the notion of directional barrier graph to model the barrier coverage formation problem. We prove that the minimum number of mobile sensors required to form a barrier with stationary sensors is the length of the shortest path from the source node to the destination node on the directional barrier graph. We then formulate the problem of minimizing the cost of moving mobile sensors to fill in the gaps on the shortest path as a minimum cost bipartite assignment problem, and solve it in polynomial time using the Hungarian algorithm. Both analytical and experimental studies demonstrate the effectiveness of the proposed algorithm.

Coverage estimation is one of the fundamental issues in many applications of sensor networks. Coverage estimation in visual sensor networks (VSNs) is more challenging than in conventional 1-D scalar sensor networks (SSNs) due to the directional sensing characteristic of cameras and the existence of visual occlusions in crowded environments. Moreover, deployment of heterogeneous visual sensors and existence of heterogeneous targets in the sensing field makes the coverage estimation problem even more challenging. In this paper, we study the coverage estimation problem in heterogeneous VSNs. We first investigate into a new target detection model, referred to as the "certainty-based target detection" as compared to the traditional "occupancy-based target detection" to facilitate the formulation of the visual coverage estimation. By adopting the new target detection model, we then derive the closed-form solution for the visual coverage estimation problem in heterogeneous VSNs. Our formulation also allows us to take both the presence of visual occlusions and boundary effect into consideration. Results from simulation validate the theoretical formulation, especially when the boundary effect is considered.

This paper presents a new hybrid dimensionality reduction method to seek projection through optimization of both structural risk (supervised criterion) and data independence (unsupervised criterion). Classification accuracy is used as a metric to evaluate the performance of the method. By minimizing the structural risk, projection originated from the decision boundaries directly improves the classification performance from a supervised perspective. From an unsupervised perspective, projection can also be obtained based on maximum independence among features (or attributes) in data to indirectly achieve better classification accuracy over more intrinsic representation of the data. Orthogonality interrelates the two sets of projections such that minimum redundancy exists between the projections, leading to more effective dimensionality reduction. Experimental results show that the proposed hybrid dimensionality reduction method that satisfies both criteria simultaneously provides higher classification performance, especially for noisy data sets, in relatively lower dimensional space than various existing methods.

Coverage estimation is one of the fundamental problems in sensor networks. Coverage estimation in visual sensor networks (VSNs) is more challenging than in conventional 1-D (omnidirectional) scalar sensor networks (SSNs) because of the directional sensing nature of cameras and the existence of visual occlusion in crowded environments. This article represents a first attempt toward a closed-form solution for the visual coverage estimation problem in the presence of occlusions. We investigate a new target detection model, referred to as the certainty-based target detection (as compared to the traditional uncertainty-based target detection ) to facilitate the formulation of the visual coverage problem. We then derive the closed-form solution for the estimation of the visual coverage probability based on this new target detection model that takes visual occlusions into account. According to the coverage estimation model, we further propose an estimate of the minimum sensor density that suffices to ensure a visual K-coverage in a crowded sensing field. Simulation is conducted which shows extreme consistency with results from theoretical formulation, especially when the boundary effect is considered. Thus, the closed-form solution for visual coverage estimation is effective when applied to real scenarios, such as efficient sensor deployment and optimal sleep scheduling.

Feature points are low-level image features representing meaningful image regions and ideal candidates for feature-based image representation, and feature point detection is an essential pre-processing step for high-level computer vision tasks. Existing feature detection algorithms are either computationally intensive (multi-scale detectors) or sensitive to scale variations (single-scale detectors). In this paper, we propose a computationally efficient multi-scale corner detector based on Discrete Wavelet Transform (DWT). We use non-redundant DWT coefficients to build a corner strength map at each scale in a data-compact way and upsample these maps by the Gaussian kernel interpolation to the original image size. By taking the summation of these maps, a corner strength measure is formed. We propose a new scale selection method that utilizes a Gaussian kernel convolution to measure the corner distribution in the vicinity of every corner point. In addition, the so-called “Polarized Gaussian” kernels are introduced to achieve rotational invariance. The high efficiency of the proposed corner detector is shown through both computational complexity analysis and accuracy analysis.

Hyperspectral images consist of large number of spectral bands but many of which contain redundant information. Therefore, band selection has been a common practice to reduce the dimensionality of the data space for cutting down the computational cost and alleviating from the Hughes phenomenon. This paper presents a new technique for band selection where a sparse representation of the hyperspectral image data is pursued through an existing algorithm, K-SVD, that decomposes the image data into the multiplication of an overcomplete dictionary (or signature matrix) and the coefficient matrix. The coefficient matrix, that possesses the sparsity property, reveals how importantly each band contributes in forming the hyperspectral data. By calculating the histogram of the coefficient matrix, we select the top K bands that appear more frequently than others to serve the need for dimensionality reduction and at the same time preserving the physical meaning of the selected bands. We refer to the proposed band selection algorithm based on sparse representation as SpaBS. Through experimental evaluation, we first use synthetic data to validate the sparsity property of the coefficient matrix. We then apply SpaBS on real hyperspectral data and use classification accuracy as a metric to evaluate its performance. Compared to other unsupervised band selection algorithms like PCA and ICA, SpaBS presents higher classification accuracy with a stable performance.

In this paper, we present r-kernel, an operating system kernel foundation specifically designed to improve software reliability in networked embedded systems. The key novelty of r-kernel lies in that it exploits the time dimension of software execution to improve robustness. Specifically, r-kernel keeps track of the execution of applications through checkpoints. If one application has been determined to have failed, r-kernel performs rollback operations to restore its state to one of those checkpoints created earlier. For the second round of operation, r-kernel provides a safe mode environment to avoid triggering the same bugs. Finally, if the whole system has crashed, r-kernel relies on watchdog timers to reset the node, and develops a technique called past-run trace reconstruction to locate and report the thread that had caused the system failure. We have implemented r-kernel based on the LiteOS operating system kernel running on the popular MicaZ platform. We demonstrate that it achieves the desired goals above with acceptable overhead.

With the development of social networks, it has been increasingly easier to make friends on the Internet. However, it may not be as easy to automatically find a friend with "similar interests". In this paper, we develop a novel system that allows users with similar interests to be quickly introduced based on the similarity of pictures they took. A real online system, named Friendbook, is implemented on a smartphone network. Due to the limited resources on a smartphone as well as privacy issues, instead of directly comparing the original pictures for similarity measure, Friendbook uses "feature-based" picture comparison. By comparing features extracted from pictures taken by people who want to make friends, their similarity in interests can be automatically inferred based on the content of these pictures. We refer to friends made through Friendbook as "S-friend" for "Semantic-friend". The system also demonstrates the difference between S-friend matching with geographic-based G-friend matching.

Quickest detection schemes are geared toward detecting a change in the state of a data stream or a real-time process. Classical quickest detection schemes invariably assume knowledge of the pre-change and post-change distributions that may not be available in many applications. In this paper, we present a distribution free nonparametric quickest detection procedure based on a novel distance measure, referred to as the Q-Q distance calculated from the Q-Q plot, for detection of distribution changes. Through experimental study, we show that the Q-Q distance-based detection procedure presents comparable or better performance compared to classical parametric and other nonparametric procedures. The proposed procedure is most effective when detecting small changes.

This paper presents an effective dimensionality reduction method based on support vector machine. By utilizing mapping vectors from support vector machine for dimensionality reduction purpose, we obtain features which are computationally efficient, providing high classification accuracy and robustness especially in noisy environment. These characteristics are acquired from the generalization capability of support vector machine by minimizing the structural risk. To further reduce dimensionality, this paper introduces the redundancy removal process based on an asymmetric decor relation measure with kernel function. Experimental results show that the proposed dimensionality reduction method provides the most appropriate trade off between classification accuracy and robustness in relatively low dimensional space.

This paper studies decentralized quickest detection schemes that can be deployed in a sensing environment where data streams are simultaneously collected from multiple channels located distributively to jointly support the detection. Existing decentralized detection approaches are largely parametric that require the knowledge of pre-change and post-change distributions. In this paper, we first present an effective nonparametric detection procedure based on Q-Q distance measure. We then describe two implementations schemes, binary quickest detection and local decision fusion by majority voting, that realize decentralized nonparametric detection. Experimental results show that the proposed method has a comparable performance to the parametric CUSUM test in binary detection. Its decision fusion-based implementation also outperforms the other three popular fusion rules under the parametric framework.

Blackouts in our daily life can be disastrous with enormous economic loss. Blackouts usually occur when appropriate corrective actions are not effectively taken for an initial contingency, resulting in a cascade failure. Therefore, it is critical to complete those tasks that are running power grid computing algorithms in the energy management system (EMS) in a timely manner to avoid blackouts. This problem can be formulated as guaranteeing end-to-end deadlines in a distributed real-time embedded (DRE) system. However, existing work in power grid computing runs those tasks in an open-loop manner, which leads to poor guarantees on timeliness thus a high probability of blackouts. Furthermore, existing feedback scheduling algorithms in DRE systems cannot be directly adopted to handle with significantly different timescales of power grid computing tasks. In this paper, we propose a hierarchical control solution to guarantee the deadlines of those tasks in EMS by grouping them based on their characteristics. Our solution is based on well-established control theory for guaranteed control accuracy and system stability. Simulation results based on a realistic workload configuration demonstrate that our solution can guarantee timeliness for power grid computing and hence help to avoid blackouts.

This paper presents a hybrid feature extraction framework based on two diverse optimization problems in aspects of risk and independence to extract features for higher classification performance. The risk minimization as a supervised approach pursues maximum generalization capability among data to directly improve classification performance, whereas the independence maximization process as an unsupervised method projects data onto a space which satisfies maximum independence to indirectly achieve better classification accuracy. Due to the direct and indirect relationship of risk minimization and independence maximization toward classification accuracy improvement, it is expected that features from the hybrid framework simultaneously satisfying both risk and independence criteria would result in the classification performance better than using either criterion. Experimental results show that the proposed hybrid framework provides higher classification performance than various existing feature extractors.

Distributed wireless sensor networks consist of energy-constrained sensor nodes that may be deployed in large numbers in order to monitor a given area. In such densely deployed environments, multiple transmissions can lead to collisions resulting in lost packets and energy wastage due to retransmissions. These networks also feature significant redundancy since nodes close to each other often sense similar data. Therefore, it may be adequate to utilize only a subset of data captured by the network. In this paper, high-energy subsets of the nodes are selected in a manner that coverage and connectivity are consistently achieved. The working subsets are changed over time after predetermined durations. A framework using concepts from spatial statistics is developed as an approach to selecting the subset of sensors. For example, an attempt at determining the correlation distance of a sensor field in the absence of real sensor data is made. Simulation results show that the algorithm is robust and retains certain level of redundancy. The ability of the algorithm to extend network lifetime is shown and the redundancy provided by the subsets selected is analyzed along with the fault tolerance provided. Conclusions regarding the flexibility and application scenarios of the algorithm are drawn.

Because of their unique capabilities, terahertz waves have demonstrated great potential in the concealed objects detection. However, the limitations in terahertz wave generation/detection devices and the disturbance from the pervasive water molecule absorption have hindered its wide deployment. Terahertz image reconstruction refers to the technique that extracts useful information from raw data to produce a meaningful image. In this paper, we present a new perspective of this technique. By defining the term differential spectrum as the difference between the transmitted spectrum and the received spectrum of the terahertz wave, we model the overall absorption effect as a linear combination of the differential spectrum of individual substance. Hence, a linear unmixing algorithm can be used to separate the overlapped substances represented by their weights, producing several image planes associated with each of the substances in the image scene. Experimental results have shown that the proposed algorithm can generate cleaner and sharper images compared to existing approaches. In addition, the object recognition process is inherent in the unmixing procedure thus making object detection and recognition easier.

Feature extraction and classification are two intertwined components in pattern recognition. Our hypothesis is that for each type of target, there exists an optimal set of features in conjunction with a specific classifier, which can yield the best performance in terms of classification accuracy using least amount of computation, measured by the number of features used. In this paper, our study is in the context of an application in wireless sensor networks (WSNs). Due to the extremely limited resources on each sensor platform, the decision making is prune to fault, making sensor fusion a necessity. We present a concept of dynamic target classification in WSNs. The main idea is to dynamically select the optimal combination of features and classifiers based on the ldquoprobabilityrdquo that the target to be classified might belong to a certain category. We use two data sets to validate our hypothesis and derive the optimal combination sets by minimizing a cost function. We apply the proposed algorithm to a scenario of collaborative target classification among a group of sensors in WSNs. Experimental results show that our approach can significantly reduce the computational time while at the same time, achieve better classification accuracy, compared with traditional classification approaches, making it a viable solution in practice.

An intrusion detection system (IDS) detects illegal manipulations of computer systems. In intrusion detection systems, feature reduction, including feature extraction and feature selection, plays an important role in a sense of improving classification performance and reducing the computational complexity. Feature reduction is even more important when online detection, which means less computational power and fast real time delivery compared with offline detection, is needed. In this paper, independent component analysis approach is applied to feature extraction in online network intrusion detection problem. We use the KDD Cup 99 data and try to reduce its 41 features such that significant less number of features would be fed into kNN and SVM classifiers. Also, a decision fusion mathod is employed to aggregate the results from multiple classifiers to achieve higher accuracy.

Real-time information dissemination is of increasing importance to our society. Existing work mainly focuses on delivering information from sources to sinks in a timely manner based on established subscriptions, with the assumption that those subscriptions are persistent. However, the bottleneck of many real-time information dissemination systems is actually the matching process to continuously reevaluate such subscriptions between numerous sources and numerous sinks, in response to dynamically varying information attributes at runtime. In this paper, we propose a feedback controller to adaptively meet the response time constraints on metadata matching in an example information dissemination system. Our controller features a rigorous design based on well-established feedback control theory for guaranteed control accuracy and system stability. Empirical results on a physical test-bed demonstrate that our controller outperforms both an open-loop solution and a typical heuristic solution, by having more accurate control and better system quality of service.

The developments in the field of wireless sensor networks (WSNs) have been accompanied by a paradigm shift from the layered protocol design to a cross layer design, which has shown its promise in effectively preserving energy, the most constraint resource in sensor networks. The proposed cross layer protocol, XLRP, explores an efficient routing strategy based on the application layer information along with the capabilities of the physical layer. Protocol design in wireless sensor networks have always resorted to low transmitter power levels as an energy efficient strategy with minimum interference. The proposed routing algorithm substantiates on switching transmission power levels based on the volume of the data being transmitted as an energy efficient mechanism. The cross layer design proposes a back-off mechanism by switching OFF unintended receivers based on the power of the received radio signal. In addition, the protocol reduces control message exchange by piggy-backing information and extracting information from packets received by unintended receivers. The effectiveness of the proposed protocol is demonstrated through simulation in ns2. The concept of operating the transmitter at various power levels is shown to be an energy efficient, scalable approach with a rational throughput.

This paper presents a procedure for calculating multiplicative inverses Modulo 2m, based on a novel mathematical approach. The procedure is suitable for software implementation on a general-purpose processor. When counting the total number of word-level processor multiplications, the computational effort involved in calculating a multiplicative inverse is 2/3 that of a single multiplication of m-bit values, in addition to a few word-level multiplications. For standard processor word sizes, the number of these additional multiplications does not exceed 12. This introduces a clear advantage of the proposed method when compared to other known methods presented in the literature.

This paper re-investigates the physical image formation process leading to a new interpretation of the classic image restoration problem from a blind source separation (BSS) perspective. The observed distorted image is considered as a linear combination of a set of shifted version of the point spread function (PSF) with the weight coefficients determined by the actual image. The new interpretation brings two immediate benefits to the practice of image restoration. First, we can utilize the rich set of BSS methods to solve the blind image restoration problem. Second, the new formulation in terms of matrix product has the equivalent merit as the conventional matrix-vector notation in theoretical study of restoration algorithms. We develop a smoothness and block-decorrelation constrained nonnegative matrix factorization method (termed CNMF) to blindly recover both the PSF and the actual image. The experimental results compared to one of the state-of-the-art methods demonstrate the merit of the proposed approach.

This paper presents a blind source separation method to unmix highly mixed hyperspectral data, i.e., each pixel is a mixture of responses from multiple materials and no pure pixels are present in the image due to large sampling distance. The algorithm introduces a minimum volume constraint to the standard non-negative matrix factorization (NMF) formulation, referred to as the minimum volume constrained NMF (MVC-NMF). MVC-NMF explores two important facts: first, the spectral data are non-negative; second, the constituent materials occupy the vertices of a simplex, and the simplex volume determined by the actual materials is the minimum among all possible simplexes that circumscribe the data scatter space. The experimental results based on both synthetic mixtures and a real image scene demonstrate that the proposed method outperforms several state-of-the-art approaches.

Endmember extraction is a process to identify the hidden pure source signals from the mixture. In the past decade, numerous algorithms have been proposed to perform this estimation. One commonly used assumption is the presence of pure pixels in the given image scene, which are detected to serve as endmembers. When such pixels are absent, the image is referred to as the highly mixed data, for which these algorithms at best can only return certain data points that are close to the real endmembers. To overcome this problem, we present a novel method without the pure-pixel assumption, referred to as the minimum volume constrained nonnegative matrix factorization (MVC-NMF), for unsupervised endmember extraction from highly mixed image data. Two important facts are exploited: First, the spectral data are nonnegative; second, the simplex volume determined by the endmembers is the minimum among all possible simplexes that circumscribe the data scatter space. The proposed method takes advantage of the fast convergence of NMF schemes, and at the same time eliminates the pure-pixel assumption. The experimental results based on a set of synthetic mixtures and a real image scene demonstrate that the proposed method outperforms several other advanced endmember detection approaches

Due to the wide existence of mixed pixels, the derivation of constituent components (endmembers) and their fractional proportions (abundances) at the subpixel scale has been given a lot of attention. The entire process is often referred to as mixed-pixel decomposition or spectral unmixing. Although various algorithms have been proposed to solve this problem, two potential issues still need to be further investigated. First, assuming the endmembers are known, the abundance estimation is commonly performed by employing a least-squares error criterion, which, however, makes the estimation sensitive to noise and outliers. Second, the mathematical intractability of the abundance non-negative constraint results in computationally expensive numerical approaches. In this paper, we propose an unsupervised decomposition method based on the classic maximum entropy principle, termed the gradient descent maximum entropy (GDME), aiming at robust and effective estimates. We address the importance of the maximum entropy principle for mixed-pixel decomposition from a geometric point of view and demonstrate that when the given data present strong noise or when the endmember signatures are close to each other, the proposed method has the potential of providing more accurate estimates than the popular least-squares methods (e.g., fully constrained least squares). We apply the proposed GDME to the subject of unmixing multispectral and hyperspectral data. The experimental results obtained from both simulated and real images show the effectiveness of the proposed method.

This paper describes the design and implementation of a distributed self-deployment algorithm in wireless visual sensor networks. The algorithm is tested on an actual mobile sensor platform (MSP) testbed, and a centralized version is also discussed for performance evaluation and comparison. Both algorithms are evaluated for speed and accuracy. Structured marker detection provided by the ARToolKit is employed for localization of the nodes. Experimental results show that localization using these types of markers has an accuracy of about 96% in ideal lighting conditions, and the proposed self-deployment algorithms perform as desired

The technique of multispectral filter array (MSFA) is a multispectral extension of the widely deployed color filter array (CFA), which uses single chip sensors and subsequent interpolation strategies to produce full color images. However, multispectral demosaicking presents unique challenges to traditional CFA demosaicking algorithms which cannot be directly extended to deal with various filter arrays with different numbers of spectral bands or different spatial patterns within each band. In addition, the more spectral bands involved, the more sparse each band samples the image plane, and the less information we can utilize when performing the interpolation. In this paper, we study a generic MSFA demosaicking method, which follows a binary tree structure to determine the order according to which different spectral bands and different pixel locations within each band are progressively interpolated, making the edge information utilized more effectively. The proposed method is demonstrated to outperform three traditional interpolation techniques as well as three advanced CFA demosaicking methods using two performance measures, the root mean square error (RMSE) for reconstruction fidelity and the classification accuracy for target recognition performance.

One hurdle involved in remote sensing imagery analysis is the wide existence of mixed pixels, whose footprints cover more than one type of ground materials. The analysis of mixed pixels demands subpixel methods to identify the ground components and infer their relative proportions, a process referred to as spectral unmixing. This paper presents a new approach to mixed pixel analysis, termed thermodynamic energy minimization (TDEM) method. The system of spectral unmixing is considered as an open information system with the measured mixed pixel as input and relative proportions as output. To find the optimal solution at the equilibrium state, we formulate an optimization problem by minimizing the Helmholtz free energy of the information system, which is derived by applying the classical maximum entropy principle to the closed system consisting of both the information system and its surrounding environment. The experimental results based on synthetic images show the effectiveness of the proposed method.

In this paper, we present a multi-layered, middleware-driven, multi-agent, interoperable architecture for distributed sensor networks, that bridges the gap between the programmable application layer consisting of software agents and the physical layer consisting of extremely small, low power devices that combine programmable computing with sensing, tracking and wireless communication capabilities. We develop an energy-efficient approach for collaborative processing among multiple sensor nodes using a mobile-agent-based computing model. Unlike the traditional client/server-based model where each sensor node sends data to the sink, in this model the sink/base-station deploys mobile agents that migrate from node to node following a certain itinerary and fuses the information/data locally at each node. This way, the intelligence is distributed throughout the network edge and communication cost is reduced to make the sensor network energy-efficient. Simulation results show that the mobile-agent-based approach is advantageous for large distributed sensor network environment

Distributed computing paradigm plays an important and fundamental role in facilitating collaboration among sensor nodes. The mobile agent computing paradigm has many benefits in the context of sensor networks. However, the most challenging problem in mobile agent based computing, the design of mobile agent itinerary, is left unanswered. An improper design of itinerary (or route) of mobile agent migration can largely deteriorate the performance of collaborative processing. In this paper, we study the key problem of deriving mobile agent itinerary for collaborative processing, especially the dynamic mobile agent planning since it is more suitable for the wireless sensor networks. This paper presents two dynamic itinerary planning algorithms, the dynamic and the predictive dynamic approaches. We design three metrics (energy consumption, network lifetime, and the number of hops ) and use simulation tools to quantitatively measure the performance of different itinerary planning algorithms. Simulation results show considerable improvement over the dynamic itinerary approach using the predictive dynamic itinerary algorithm

The technology of color filter arrays (CFA) has been widely used in the digital camera industry since it provides several advantages like low cost, exact registration, and strong robustness. The same motivations also drive the design of multispectral filter arrays (MSFA), in which more than three spectral bands are used. Although considerable research has been reported to optimally reconstruct the full-color image using various demosaicking algorithms, studies on the intrinsic properties of these filter arrays as well as the underlying design principles have been very limited. Given a set of representative spectral bands, the design of an MSFA involves two issues: the selection of tessellation mechanisms and the arrangement/layout of different spectral bands. We develop a generic MSFA generation method starting from a checkerboard pattern. We show, through case studies, that most of the CFAs currently used by the industry can be derived as special cases of MSFAs generated using the generic algorithm. The performance of different MSFAs are evaluated based on their intrinsic properties, namely, the spatial uniformity and the spectral consistency. We design two metrics, static coefficient and consistency coefficient, to measure these two parameters, respectively. The experimental results demonstrate that the generic algorithm can generate optimal or near-optimal MSFAs in both the rectangular and the hexagonal domains.

In this paper, we extend the idea of using mosaicked color filter array (CFA) in color imaging, which has been widely adopted in the digital color camera industry, to the use of multispectral filter array (MSFA) in multispectral imaging. The filter array technique can help reduce the cost, achieve exact registration, and improve the robustness of the imaging system. However, the extension from CFA to MSFA is not straightforward. First, most CFAs only deal with a few bands (3 or 4) within the narrow visual spectral region, while the design of MSFA needs to handle the arrangement of multiple bands (more than 3) across a much wider spectral range. Second, most existing CFA demosaicking algorithms assume the fixed Bayer CFA and are confined to properties only existed in the color domain. Therefore, they cannot be directly applied to multispectral demosaicking. The main challenges faced in multispectral demosaicking is how to design a generic algorithm that can handle the more diversified MSFA patterns, and how to improve performance with a coarser spatial resolution and a less degree of spectral correlation. In this paper, we present a binary tree based generic demosaicking method. Two metrics are used to evaluate the generic algorithm, including the root mean-square error (RMSE) for reconstruction performance and the classification accuracy for target discrimination performance. Experimental results show that the demosaicked images present low RMSE (less than 7) and comparable classification performance as original images. These results support that MSFA technique can be applied to multispectral imaging with unique advantages.

Dynamic formation of node clusters is inherently embedded in a wide range of emerging wireless sensor network (WSN) applications. It is expected that security will play a key role in the design and successful deployment of these, as well as many other, applications. The ad-hoc nature and unique power-constraint characteristics of WSN suggest that a prerequisite for achieving security is the ability to encrypt and decrypt confidential data among an arbitrary set of sensor nodes. Consequently, the nodes are required to generate a joint secret key. Elliptic curve cryptography (ECC) has emerged as a suitable public key cryptographic foundation for WSN. This paper describes a pragmatic ECC-based methodology for self-certified group key generation in ad hoc clusters of sensor nodes. A novel load-balancing technique and chained data exchange yield reduced overall communications and facilitate an efficient distribution of the computational effort involved.

This paper studies the problem of self-deployment of heterogeneous mobile sensors using biologically-inspired principles and methodologies. The initial sensor deployment is assumed to be random, based on which two interrelated issues are investigated: the design of an optimal placement pattern of heterogeneous sensor platforms and the self configuration from the initial random state to the optimal state through intelligent sensor movement. We first develop an optimal placement algorithm based on the mosaic technique inspired by the retina mosaic pattern widely observed in both human and many animal visual systems. Different types of mobile sensors are organized into a mosaic pattern to maximize sensing coverage and to reduce network cost. Secondly, in order to converge to the optimal state, we investigate the swarm intelligence (SI)-based sensor movement strategy with the assistance of local communications, through which the randomly deployed sensors can self-organize themselves to reach the optimal placement state. The proposed algorithm is compared with the random movement and the SI-based method without direct communication using performance metrics such as sensing coverage, redundancy, convergence time, and energy consumption. Simulation results are presented to demonstrate the effectiveness of the mosaic placement and the SI-based movement with local communication.

We present an innovative MAC protocol (Q-MAC) that minimizes the energy consumption in multi-hop wireless sensor networks (WSNs) and provides quality of service (QoS) by differentiating network services based on priority levels. The priority levels reflect application priority and the state of system resources, namely residual energy and queue occupancies. The Q-MAC utilizes both intra-node and inter-node arbitration. The intra-node packet scheduling is a multiple queuing architecture with packet classification and weighted arbitration. We also introduce the power conservation MACAW (PC-MACAW) - a power-aware scheduling mechanism that, together with the loosely prioritized random access (LPRA) algorithm, govern the inter-node scheduling. Performance evaluation are conducted between Q-MAC and S-MAC with respect to two performance metrics: energy consumption and average latency. Simulation results indicate that the performance of the Q-MAC is comparable to that of the S-MAC in non-prioritized traffic scenarios; when packets with different priorities are present, Q-MAC supiors in average latency differentiation between the classes of service, while maintaining the same energy level as that of S-MAC

In this paper, we study the multiple target detection problem in sensor networks. Due to the unique features possessed by sensor networks, the multiple target detection approach has to be energy-efficient and bandwidth-efficient. The existing centralized solutions cannot satisfy these requirements. We present a progressive decentralized detection approach based on the classic Bayesian source number estimation algorithm. The progressive approach is realized by a mobile agent framework, where instead of each sensor sending raw data to a central unit, each sensor processes data locally and a mobile agent is dispatched from the central unit and migrates in the network updating the estimation progressively. Experimental results show that this approach can achieve progressive accuracy with the migration of mobile agent and reduce data transmission dramatically.

The technology of color filter arrays (CFA) has been widely used in the digital camera industry since it provides several advantages like low cost, exact registration, and strong robustness. The same motivations also drive the design of multi-spectral filter arrays (MSFA), in which more than three color bands are used (e.g. visible and infrared). Although considerable research has been reported to optimally reconstruct the full-color image using various interpolation algorithms, studies on the intrinsic properties of these filter arrays as well as the underlying design principles have been very limited. In this paper, we identify the properties a CFA should possess and extend the design philosophy to MSFA. Based on these discussions, we develop a generic MSFA generation method starting from a checkerboard pattern with both rectangular and hexagonal tessellations. By manipulating this pattern through a combination of decomposition and subsampling steps, we can generate MSFAs that satisfy all the design requirements. We show, through case studies, that most of the CFAs currently used by the industry can be derived as special cases. To evaluate the performance of MSFAs, we design a metric, referred as the static coefficient (SC), to measure the uniformity of MSFAs.

Independent component analysis (ICA) is a technique that extracts independent source signals by searching for a linear or nonlinear transformation which minimizes the statistical dependence between components. ICA has been used in a variety of signal processing applications including dimensionality reduction in hyperspectral image (HSI) analysis. Due to the computation complexities and convergence rates, ICA is very time-consuming for high volume or dimension data set like hyperspectral images. Hardware implementation provides not only an optimal parallelism environment, but also a potential faster and real-time solution. This work synthesizes a parallel ICA (pICA) algorithm on field programmable gate array (FPGA). In the proposed implementation method, the pICA is partitioned into three temporally independent functional modules, and each of which is synthesized individually with several ICA-related reconfigurable components (RCs) that are developed for reuse and retargeting purpose. All modules are then integrated into a design and development environment for performing many subtasks such as FPGA synthesis, optimization, placement and routing. In a case study, we synthesize the pICA algorithm for hyperspectral image dimensionality reduction on the pilchard reconfigurahle computing platform embedded with Xilinx: VIRTEX V1000E. The FPGA executes at the maximum frequency of 20.161 MHz, and the pilchard board transfers data directly with CPU on the 64-bit memory bus at the maximum frequency of 133MHz. The performance comparisons between the proposed and another two ICA-related FPGA implementations show that the proposed FPGA implementation of pICA has potential in performing complicated algorithms on large volume data sets.

The problem of computing a route for a mobile agent that incrementally fuses the data as it visits the nodes in a distributed sensor network is considered. The order of nodes visited along the route has a significant impact on the quality and cost of fused data, which, in turn, impacts the main objective of the sensor network, such as target classification or tracking. We present a simplified analytical model for a distributed sensor network and formulate the route computation problem in terms of maximizing an objective function, which is directly proportional to the received signal strength and inversely proportional to the path loss and energy consumption. We show this problem to be NP-complete and propose a genetic algorithm to compute an approximate solution by suitably employing a two-level encoding scheme and genetic operators tailored to the objective function. We present simulation results for networks with different node sizes and sensor distributions, which demonstrate the superior performance of our algorithm over two existing heuristics, namely, local closest first and global closest first methods.

In sensor networks, collaborative processing between multiple sensor nodes is essential in order to complement for each others sensing capability, tolerate faults, and provide reliable information. The client/server-based paradigm is typical for distributed processing. However, it is not the most efficient in the context of sensor networks. In this paper, we present a mobile agent-based paradigm to carry out collaborative processing, where instead of each sensor node sending local information to a processing center, as is typical in the client/server-based computing, the processing code is moved to the sensor nodes through mobile agents. This approach has great potential in providing energy-efficient and scalable collaborative processing with low latency. We design two metrics (execution time and energy consumption) and use simulation tools to quantitatively measure the performance of different computing models in collaborative processing. Experimental results show that the mobile agent paradigm performs much better when the number of nodes is large while the client/server paradigm is advantageous when the number of nodes is small. Based on this result, we develop a cluster-based hybrid computing paradigm to combine the advantages of both paradigms. We analyze two different scenarios in hybrid computing and simulation results show that there is always one scenario that performs better than either the client/server- or mobile-agent-based paradigm.

In mobile-agent-based distributed sensor networks (MADSNs), instead of moving data from an individual sensor node to a processing center as a typical scenario in the client/server-based computing, mobile agents on that carry the executable code are dispatched from the processing center to the sensor nodes and process data locally. Because of the complicated behavior of the mobile agents, there has not been much work done in modeling and simulation. In this paper, a Generalized Stochastic Petri Net (GSPN) is used to model the mobile agents in DSNs. GSPN is a very popular modeling tool for systems that feature concurrency, synchronization and randomness. One of the most challenging problems in GSPN modeling is to design a mechanism for breaking the transition conflicts. This paper presents a random transition selector based on joint entropy and a so-called "rolling rocks random (R3) selector" to break conflicts among immediate transitions. The GSPN model based on this transition selector is synthesized on a Xilinx Virtex 1000E Field Programmable Gate Array (FPGA) using reconfigurable components. Simulation results show that the proposed transition selector performs better than the commonly used random selector.

In this paper, we develop an energy-efficient, fault-tolerant approach for collaborative signal and information processing (CSIP) among multiple sensor nodes using a mobile-agent-based computing model. In this model, instead of each sensor node sending local information to a processing center for integration, as is typical in client/server-based computing, the integration code is moved to the sensor nodes through mobile agents. The energy efficiency objective and the fault tolerance objective always conflict with each other and present unique challenge to the design of CSIP algorithms. In general, energy-efficient approaches try to limit the redundancy in the algorithm so that minimum amount of energy is required for fulfilling a certain task. On the other hand, redundancy is needed for providing fault tolerance since sensors might be faulty, malfunctioning, or even malicious. A balance has to be struck between these two objectives. We discuss the potential of mobile-agent-based collaborative processing in providing progressive accuracy while maintaining certain degree of fault tolerance. We evaluate its performance compared to the client/server-based collaboration from perspectives of energy consumption and execution time through both simulation and analytical study. Finally, we take collaborative target classification as an application example to show the effectiveness of the proposed approach.

The most challenging problem in target detection and classification is the extraction of a robust feature vector which can effectively represent a specific type of target. The use of the seismic signals in unattended ground sensor systems brings new challenges to the problem because of the complexity of the seismic waves and their highly dependency on the underlying geology. This paper proposes a new feature extraction algorithm - spectral statistics and wavelet coefficients characterization (SSWCC). SSWCC extracts a feature vector from both the frequency and the time-frequency domain analysis of the seismic signals, including the spectrum, the power spectral density (PSD) and the wavelet coefficients. The SSWCC algorithm is designed for real-time applications, and has shown its robustness and effectiveness through a series of experiments. Extensive performance evaluation is conducted to derive the optimal configuration of the different parameters. The overall classification accuracy can reach as high as 90.

Target classification using distributed sensor arrays remains a challenging problem due to the non-stationarity of target signatures, large geographical area coverage of sensor arrays, and the requirements of time-critical and reliable information delivery. In this paper, we develop an algorithm to derive effective and stable features from both the frequency and the time-frequency domains of the acoustic signals. A modified data fusion algorithm for distributed sensor arrays is also developed in order to integrate the classification results from different sensors and provide fault-tolerance. By using data fusion, the accuracy of the classification can be increased by as many as 50%.

We propose a new approach to steganography for information hiding in digital media. We refer to it as blind consistency-based steganography (BCBS). Our main concerns in designing BCBS is high imperceptibility, high security level, and large capacity. Steganography differs from digital watermarking because both the information and the very existence of the information are hidden. Even though the discussion focuses on digital images, BCBS can be applied to any digital media. While existing steganographic approaches all have different advantages for specific applications, they also suffer a similar problem: such systems are unable to handle subterfuge attacks, i.e., they cannot deal with the opponents who not only detect a message, but also render it useless, or even worse, modify it to the opponent's favor. The breakthrough of BCBS is that it not only decodes the message exactly, it also detects if the message has been tampered without using any extra error correction. The detection is integrated into the process of decoding. Another advantage of BCBS is the "blindness" of the decoding process, meaning the decoding can be operated without access to the cover image. The encoding and decoding process are detailed in the paper, along with the analysis on imperceptibility, security, and capacity. Experimental results are provided as well.

The emergence of mobile ad hoc sensor networks has brought new challenges to traditional network design. This paper compares the performance of two distributed computing paradigms, the client/server-based paradigm and the mobile-agent-based paradigm, through mathematical modeling and simulation. Previous works have shown that the mobile-agent-based paradigm is more appropriate to handle computations in ad hoc sensor networks. However, no simulation work has been done to quantitatively measure the performance. This paper first describes how computing is accomplished in the mobile-agent-based paradigm. It then presents a modified mathematical model and uses the execution time as a metric to measure the performance. Eight experiments are designed to show the effect of different parameters to the performance of the paradigms. Experimental results show that in the context of mobile ad hoc sensor networks with hundreds or even thousands of nodes, unreliable communication links and reduced bandwidth, the mobile-agent-based computing provides solutions to low network latency and reliable data processing.

We present novel grid coverage strategies for effective surveillance and target location in distributed sensor networks. We represent the sensor field as a grid (two or three-dimensional) of points (coordinates) and use the term target location to refer to the problem of locating a target at a grid point at any instant in time. We first present an integer linear programming (ILP) solution for minimizing the cost of sensors for complete coverage of the sensor field. We solve the ILP model using a representative public-domain solver and present a divide-and-conquer approach for solving large problem instances. We then use the framework of identifying codes to determine sensor placement for unique target location, We provide coding-theoretic bounds on the number of sensors and present methods for determining their placement in the sensor field. We also show that grid-based sensor placement for single targets provides asymptotically complete (unambiguous) location of multiple targets in the grid.

We describe the use of the mobile agent paradigm to design an improved infrastructure for data integration in a distributed sensor network (DSN). We use the acronym MADSN to denote the proposed mobile-agent-based DSN. Instead of moving data to processing elements for data integration, as is typical of a client/server paradigm, MADSN moves the processing code to the data locations. This saves network bandwidth and provides an effective means for overcoming network latency, since large data transfers are avoided. Our major contributions are the use of mobile agent in DSN for distributed data integration and the evaluation of performance between DSN and MADSN approaches. We develop an enhanced multiresolution integration (MRI) algorithm where multiresolution analysis is applied at a local node before accumulating the overlap function by mobile agent. Compared to the MRI implementation in DSN, the enhanced integration algorithm saves up to 90% of the data transfer time. We develop objective functions to evaluate the performance between DSN and MADSN approaches. For a given set of network parameters, we analyze the conditions under which MADSN performs better than DSN and determine the condition under which MADSN reaches its optimum performance level.

Optimal missing data estimation algorithms including deblurring and denoising are designed to restore images captured from large CCD sensor arrays using a butting technique, where 1 to 2 columns of data are missed at the butting edge. We developed a consistency method with separable deblurring to estimate the missing data. This method converts an ill-posed restoration problem into a well-posed one by making few assumptions based on regularization theory. Under the condition that no noise is inserted, and the separable blur kernel is exactly known, the consistency method can deblur the original image and at the same time estimate the missing columns(s) exactly. However, this algorithm becomes unstable when large noise is inserted or inaccurate estimation of the blur kernel is made. Conditioning analysis is used to quantify the amount of ill condition of the blur kernel when the assumptions are relaxed to different levels, which provides a solid measurement on how stable the system will remain knowing the signal-to-noise ratio and the inaccuracy of the blur kernel estimation. Experimental results from different approaches are compared.

Anisotropic diffusion is a powerful method for image feature extraction in which blurring is allowed to occur except at edges. Mean field annealing (MFA) is an image optimization technique which is used to find the best estimation of a blurred, noisy corrupted image. We show that MFA is versatile enough to be used for image feature extraction as well. Furthermore, the two major problems in anisotropic diffusion (white noise and geometric interpretation) can be solved by few modifications of the MFA equation. Finally, experiments and results are presented.