Anomaly-Based Intrusion Detection Systems (IDSs) have been extensively researched for their ability to detect zero-day attacks. These systems establish a baseline of normal behavior using benign traffic data and flag deviations from this norm as potential threats. They generally experience higher false alarm rates than signature-based IDSs. Unlike image data, where the observed features provide immediate utility, raw network traffic necessitates additional processing for effective detection. It is challenging to learn useful patterns directly from raw traffic data or simple traffic statistics (e.g., connection duration, package inter-arrival time) as the complex relationships are difficult to distinguish. Therefore, some feature engineering becomes imperative to extract and transform raw data into new feature representations that can directly improve the detection capability and reduce the false positive rate. We propose a geometric feature learning method to optimize the feature extraction process. We employ contrastive feature learning to learn a feature space where normal traffic instances reside in a compact cluster. We further utilize H-Score feature learning to maximize the compactness of the cluster representing the normal behavior, enhancing the subsequent anomaly detection performance. Our evaluations using the NSL-KDD and N-BaloT datasets demonstrate that the proposed IDS powered by feature learning can consistently outperform state-of-the-art anomaly-based IDS methods by significantly lowering the false positive rate. Furthermore, we deploy the proposed IDS on a Raspberry Pi 4 and demonstrate its applicability on resource-constrained Internet of Things (IoT) devices, highlighting its versatility for diverse application scenarios.

Computing eigenvalue decomposition (EVD) of a given linear operator, or finding its leading eigenvalues and eigenfunctions, is a fundamental task in many machine learning and scientific simulation problems. For high-dimensional eigenvalue problems, training neural networks to parameterize the eigenfunctions is considered as a promising alternative to the classical numerical linear algebra techniques. This paper proposes a new optimization framework based on the low-rank approximation characterization of a truncated singular value decomposition, accompanied by new techniques called nesting for learning the top-$L$ singular values and singular functions in the correct order. The proposed method promotes the desired orthogonality in the learned functions implicitly and efficiently via an unconstrained optimization formulation, which is easy to solve with off-the-shelf gradient-based optimization algorithms. We demonstrate the effectiveness of the proposed optimization framework for use cases in computational physics and machine learning.

We present a novel framework for learning system design with neural feature extractors. First, we introduce the feature geometry, which unifies statistical dependence and feature representations in a function space equipped with inner products. This connection defines function-space concepts on statistical dependence, such as norms, orthogonal projection, and spectral decomposition, exhibiting clear operational meanings. In particular, we associate each learning setting with a dependence component and formulate learning tasks as finding corresponding feature approximations. We propose a nesting technique, which provides systematic algorithm designs for learning the optimal features from data samples with off-the-shelf network architectures and optimizers. We further demonstrate multivariate learning applications, including conditional inference and multimodal learning, where we present the optimal features and reveal their connections to classical approaches.

We study kernel methods in machine learning from the perspective of feature subspace. We establish a one-to-one correspondence between feature subspaces and kernels and propose an information-theoretic measure for kernels. In particular, we construct a kernel from Hirschfeld–Gebelein–Rényi maximal correlation functions, coined the maximal correlation kernel, and demonstrate its information-theoretic optimality. We use the support vector machine (SVM) as an example to illustrate a connection between kernel methods and feature extraction approaches. We show that the kernel SVM on maximal correlation kernel achieves minimum prediction error. Finally, we interpret the Fisher kernel as a special maximal correlation kernel and establish its optimality.

For signal detection tasks in wireless communications, most of the existing algorithms either ignore the co-channel interference or treat it as Gaussian noise, which may result in unsatisfactory accuracy when the interference is non-negligible and complex-distributed. When neural networks are motivated in the design of the detectors, difficulty arises in the training due to the fact that there are few accessible pilots in each packet. In this paper, we consider a data-driven detector based on multi-source transfer learning (MSTL) for signal detection in a fading channel with interference. The MSTL detector transfers channel knowledge of previous packets into the latest detection. In particular, we consider a linear combination of the pilots and historical symbols in the distribution space, and design the optimal combination coefficients based on the number of those symbols as well as distributions similarity. Numerical simulations on a Gauss-Markov flat Rayleigh fading channel with co-channel interference validate the advantages of our algorithms, compared with several existing training schemes including directly applying the fully connected deep neural network (FCDNN) and conventional linear minimum mean square error (LMMSE) detector.

Fine-grained city-scale outdoor air pollution maps provide important environmental information for both city managers and residents. Installing portable sensors on vehicles (e.g., taxis, Ubers) provides a low-cost, easy-maintenance, and high-coverage approach to collecting data for air pollution estimation. However, as non-dedicated platforms, vehicles like taxis usually prefer gathering at busy areas of a city where it is more likely to pick up riders. This leaves many parts of the city unsensed or less-sensed. In addition, due to the natural changes in a city and the movements of the vehicles, the sensed and unsensed areas change over time. Consequently, challenges of air pollution estimation with data collected by non-dedicated mobile platforms are twofold:i.data coverage is sparse;ii.data coverage changes over time. Therefore, the major research question is: how can we derive accurate and robust fine-grained field (e.g., air pollution) estimation given dynamic and sparse data collected from uncontrollable mobile sensing platforms? This paper presents adaptiveHMSS, an adaptivehybridmodel-enabledsensingsystem for fine-grained air pollution estimation with dynamic and sparse data collected from uncontrollable mobile sensing platforms, which is achieved by combining the advantages of aphysics guided modeland adata driven model. To address the challenge of sparse coverage, the physical understanding of the spatiotemporal correlation for air pollution distribution in thephysics guided modelis utilized to infer values at unsensed sparse areas. Meanwhile, thedata driven modelis adopted to estimate the air pollution influential factors (e.g., buildings) not included in thephysics guided model. To address the challenge of time-varying coverage, an adaptive model combination algorithm is designed to enable the system bias to either of the two models according to the amount of data collection and uncertainty of the model. To evaluate the system performance, we deployed 47 air pollution sensing devices on taxis and fixed locations in 2 cities for both controlled and uncontrolled experiments for over two weeks. The results show that with a resolution of$500 \;\mathrm m$by$500\;\mathrm m$by$1\;\mathrm {hour}$, our system achieves up to$3.2\times$error reduction when compared to the baseline approaches.

Distributed learning is recently an important research topic, while the information theoretic optimality of the distributed learning algorithms is often not sufficiently addressed. This paper studies the distributed learning problems such that each node observes i.i.d. samples and sends a feature function of observed samples to the central machine for decision making. Both the binary hypothesis testing in information theory and the classification problems in machine learning are considered, and the optimal error exponent and the set of optimal features are characterized. By exploiting an information theoretic framework, we show that these two problems share the same set of optimal features, from which the information theoretic optimality of some machine learning algorithms can be established. Finally, we generalize our analyses to$M$-ary distributed hypothesis testing and classification problems. A full version of this paper is accessible at: https://xiangxiangxu.com/media/documents/isit2021.pdf

In this paper, we consider the distributed hypothesis testing (DHT) problem where two nodes are constrained to transmit constant bits to a central decoder. In such cases, we show that in order to achieve the optimal error exponents, it suffices to consider the empirical distributions of observed data sequences and encode them to the transmission bits. With such a coding strategy, we develop a geometric approach in the distribution spaces to show the optimal achievable error exponents and coding scheme for the following cases: (i) both nodes can transmit $\log_{2}3$ bits; (ii) one of the nodes can transmit 1 bit, and the other node is not constrained; (iii) the joint distribution of the nodes are conditionally independent given one hypothesis. Our approach essentially reveals new potentials for characterizing the precise error exponents for DHT with general communication constraints.

Current transfer learning algorithm designs mainly focus on the similarities between source and target tasks, while the impacts of the sample sizes of these tasks are often not sufficiently addressed. This paper proposes a mathematical framework for quantifying the transferability in multi-source transfer learning problems, with both the task similarities and the sample complexity of learning models taken into account. In particular, we consider the setup where the models learned from different tasks are linearly combined for learning the target task, and use the optimal combining coefficients to measure the transferability. Then, we demonstrate the analytical expression of this transferability measure, characterized by the sample sizes, model complexity, and the similarities between source and target tasks, which provides fundamental insights of the knowledge transferring mechanism and the guidance for algorithm designs. Furthermore, we apply our analyses for practical learning tasks, and establish a quantifiable transferability measure by exploiting a parameterized model. In addition, we develop an alternating iterative algorithm to implement our theoretical results for training deep neural networks in multi-source transfer learning tasks. Finally, experiments on image classification tasks show that our approach outperforms existing transfer learning algorithms in multi-source and few-shot scenarios.

The Hirschfeld-Gebelein-Rényi (HGR) maximal correlation and the corresponding functions have been shown useful in many machine learning scenarios. In this paper, we study the sample complexity of estimating the HGR maximal correlation functions by the alternating conditional expectation (ACE) algorithm using training samples from large datasets. Specifically, we develop a mathematical framework to characterize the learning errors between the maximal correlation functions computed from the true distribution, and the functions estimated from the ACE algorithm. For both supervised and semi-supervised learning scenarios, we establish the analytical expressions for the error exponents of the learning errors. Furthermore, we demonstrate that for large datasets, the upper bounds for the sample complexity of learning the HGR maximal correlation functions by the ACE algorithm can be expressed using the established error exponents. Moreover, with our theoretical results, we investigate the sampling strategy for different types of samples in semi-supervised learning with a total sampling budget constraint, and an optimal sampling strategy is developed to maximize the error exponent of the learning error. Finally, the numerical simulations are presented to support our theoretical results.

While it is commonly believed that estimating the small singular modes for a nearly low-rank matrix requires more samples, the sample size needed is generally unclear. In this paper, we investigate this sample complexity by considering the difference between the estimation errors of estimating a matrix with or without estimating these small singular modes. Specifically, we develop a mathematical framework based on the matrix perturbation analysis to characterize the noise level of estimating small singular modes by n samples. In particular, we show that under mild assumptions on the sample noise, it requires at least n = O(η-2) samples to well estimate the singular modes with the singular value in the order of some small η. More importantly, our results are applied to the channel state estimation and Hirschfeld-Gebelein-Rényi (HGR) maximal correlation problems, from which we characterize that for how many samples, utilizing the low-rank approximation in these problems are beneficial. Finally, numerical simulations are provided to verify our results.

While the Hirschfeld-Gebelein-Rényi (HGR) maximal correlation and the Wyner common information share similar information processing purposes of extracting common knowledge structures between random variables, the relationships between these approaches are generally unclear. In this paper, we demonstrate such relationships by considering the Wyner common information in the weakly dependent regime, called ε-common information. We show that the HGR maximal correlation functions coincide with the relative likelihood functions of estimating the auxiliary random variables in ε-common information, which establishes the fundamental connections these approaches. Moreover, we extend the ε-common information to multiple random variables, and derive a novel algorithm for extracting feature functions of data variables regarding their common information. Our approach is validated by the MNIST problem, and can potentially be useful in multi-modal data analyses.

One primary focus in multimodal feature extraction is to find the representations of individual modalities that are maximally correlated. As a well-known measure of dependence, the Hirschfeld-Gebelein-Rényi (HGR) maximal correlation be-´ comes an appealing objective because of its operational meaning and desirable properties. However, the strict whitening constraints formalized in the HGR maximal correlation limit its application. To address this problem, this paper proposes Soft-HGR, a novel framework to extract informative features from multiple data modalities. Specifically, our framework prevents the “hard” whitening constraints, while simultaneously preserving the same feature geometry as in the HGR maximal correlation. The objective of Soft-HGR is straightforward, only involving two inner products, which guarantees the efficiency and stability in optimization. We further generalize the framework to handle more than two modalities and missing modalities. When labels are partially available, we enhance the discriminative power of the feature representations by making a semi-supervised adaptation. Empirical evaluation implies that our approach learns more informative feature mappings and is more efficient to optimize.

Multilabel learning, the problem of mapping each data instance to a subset of labels, appears frequently in many real-world applications. However, obtaining complete label annotation for every instance requires tremendous efforts, especially when the label set is large. As a result, multilabel learning with missing labels remains as a common challenge. Existing works either cannot handle missing labels or lack nonlinear expressiveness and scalability to large label set. In this paper, we present a novel end-to-end solution for multilabel learning with missing labels. Our algorithm, Maximal Correlation Embedding Network learns a low dimensional label embedding using an encoder-decoder architecture. It exploits label similarity through a maximal correlation regularization in the embedded label space to reduce the classification bias due to missing labels. A series of experiments on popular multilabel datasets demonstrate that our approach outperforms state of the art, both in complete data and partially observed data.

Real-time human context recognition is one of the most exciting emerging technologies in sensing nowadays. Compared with most recognition problems in machine learning, the challenge lies in the complexity and incompleteness of labels, in other words, each sample can have several label concepts simultaneously but some of them could be missing. This poster proposes an effective approach for multilabel human context recognition with signals from sensors embedded in the wearable devices. The proposed algorithm demonstrates to be very robust to incomplete labels.

In this paper, we address the issue of computing the maximal correlation functions for jointly distributed high-dimensional random variables. In such cases, the operations in the alternative conditional expectation (ACE) algorithm can only be implemented by an approximated manner, which is modeled as a variational ACE algorithm with noise. We study the computational behaviors of this algorithm, where the optimal tradeoff between the learning rate, computation accuracy, and convergence rate is investigated. In addition, we establish a connection between the variational ACE algorithm and the residual learning architecture. Our results illustrate interesting interpretations of how multi-layer residual structure benefits function learning.

It is commonly believed that the hidden layers of deep neural networks (DNNs) attempt to extract informative features for learning tasks. In this paper, we formalize this intuition by showing that the features extracted by DNN coincide with the result of an optimization problem, which we call the "universal feature selection" problem, in a local analysis regime. We interpret the weights training in DNN as the projection of feature functions between feature spaces, specified by the network structure. Our formulation has direct operational meaning in terms of the performance for inference tasks, and gives interpretations to the internal computation results of DNNs. Results of numerical experiments are provided to support the analysis.

The Hirschfeld-Gebelein-Rényi (HGR) maximal correlation has been shown useful in many machine learning scenarios. In this paper, we investigate the sample complexity problem of estimating the HGR maximal correlation functions by the alternative conditional expectation (ACE) algorithm from a sequence of training data in the asymptotic regime. Specifically, we develop a mathematical framework to characterize the eigen-decomposition of perturbed matrices, and then establish the error exponent of the learning error for the computed HGR maximal correlation functions. Our result essentially indicates the number of training samples required for estimating the HGR maximal correlation functions to a targeted accuracy by the ACE algorithm.

The surveillance of air pollution is becoming a highly concerned issue for city residents and urban administrators. Fixed air quality stations as well as mobile gas sensors have been deployed for air quality monitoring but with sparse observations over the entire temporal-spatial space. Therefore, an inference algorithm is essential for comprehensive fine-grained air pollution sensing. Conventional physically-based models can hardly be applied to all the scenarios, while pure data-driven methods suffer from sampling bias and overfitting problems. This paper presents a hybrid algorithm for air pollution inference by guiding the data learning process with physical model. The quantitative combination of knowledge from observed dataset and a discretized convective-diffusion model is performed within a multi-task learning scheme. Evaluations show that, benefited from physical guidance, our hybrid method obtains higher extrapolation ability and more robustness, achieving the same performance with 1/8 sample amount and obtaining 31.9% less error in noisy synthesized environment. In a real-world deployment in Tianjin, our algorithm outperforms the pure data-driven model with 9.69% less inference error over a 9-day PM2.5data collection.

In this paper, we formulate the generalized softmax learning (GSL) problem, as a symmetric extension of the softmax regression problem. We further study the geometric structure of GSL and demonstrate the equivalence of GSL and the original softmax regression problem. Besides, this geometric structure indicates the symmetry between a neural network and its reverse network, and the symmetric roles of the weights and feature in a neural network. Finally, we present a numerical simulation to verify these symmetry properties in neural networks.

Mobile sensing systems are deployed for urban air pollution monitoring to increase coverage over a city. However, the sampling irregularity brings great challenges for fine-grained pollution field recovery. To address this problem, we proposed a generative model based inference algorithm. By modeling the air pollution evolution and data sampling process separately, the temporal-spatial correlation of pollution field can be considered with irregular sampled data. We use a convolutional long-short term memory structure in the generative model and train it with the scattered observations from mobile sensing. Evaluations on synthesized data and a deployment in the city of Tianjin show that our algorithm accurately captures fine-grained PM2.5 pollution patterns and changes. The average inference error is 6.7μg/m3, which achieves 23.8% improvement over existing techniques.

In this paper, given the scenario of a mobile sensing system for air pollution monitoring, we aim at the cause and influence of delay effect on measurement and present a filter-based solution to calibrate the sensing data. We also validate the idea and solution by a real-data experiment. It indicates that the solution decreases deviation on spatial measurement and can be applied in mobile sensing systems to improve the sensing data quality.

Low-cost sensors are widely used to realize large-scale deployment for sensing systems. In this paper, we discuss challenges in using industrial-grade gas sensors for air quality monitoring. To overcome variation due to system errors, we present a framework for individualized calibration. Within the framework, multiple regression and interpolation methods are prepared for alternative optimization on fitting sensors' response to gas concentration.

This paper presents a hybrid adaptive particle filter (HAP) with online feedback to dynamically reconstruct high spatial-temporal resolution air pollution information from sparse vehicular based sensors. To deal with data sparsity, we apply both spatial and temporal correlation of air dispersion to reduce data dimension requirement. HAP adaptively predicts when the accumulated prediction error is low and then uses data compensation for correction whenever the prediction error becomes high. The preliminary results based on the city scale deployments with 10 taxis show that our system achieves up to 50% reduction on system errors.

According to the World Health Organization (WHO), outdoor air pollution led to an estimated 3.7 million premature deaths worldwide in 2012. To address this problem, it is necessary for both residents and city administrations to understand air quality in their immediate environment with fine-grained temporal-spatial resolution. Currently both fixed and mobile systems are used to attempt to sense the pollution field. However, they generally are expensive, cover small areas and thus result in lower accuracy.

Urban environment has significant impacts on the health of city dwellers. To understand these impacts, city planners have to obtain fine-grained environmental information, however such information is not available with traditional environmental systems. To address this problem, we present Gotcha, a taxi-based mobile sensing system for fine-grained environmental data acquisition. Gotcha utilizes taxi cabs to serve as a sensor that collects a variety of environmental information (such as concentrations of carbon-dioxide, carbon-monoxide, ozone, particulate matter, etc.). We aim to deploy our system in the city of Shenzhen on a fleet of 100 taxi cabs, and we present here our results from our initial deployment.