This paper addresses the challenge of integrating sequentially arriving data into the quantile regression framework, where the number of features may increase with the number of observations, the time horizon is unknown, and memory resources are limited. Unlike least squares and robust regression methods, quantile regression models different segments of the conditional distribution, thereby capturing heterogeneous relationships between predictors and responses and providing a more comprehensive view of the underlying stochastic structure. We employ stochastic sub-gradient descent to minimize the empirical check loss and analyze its statistical properties and regret behavior. Our analysis reveals a subtle interplay between updating iterates based on individual observations and on batches of observations, highlighting distinct regularity characteristics in each setting. The proposed method guarantees long-term optimal estimation performance regardless of the chosen update strategy. Our contributions extend existing literature by establishing exponential-type concentration inequalities and by achieving optimal regret and error rates that exhibit only short-term sensitivity to initialization. A key insight from our study lies in the refined statistical analysis showing that properly chosen stepsize schemes substantially mitigate the influence of initial errors on subsequent estimation and regret. This result underscores the robustness of stochastic sub-gradient descent in managing initial uncertainties and affirms its effectiveness in sequential learning settings with unknown horizons and data-dependent sample sizes. Furthermore, when the initial estimation error is well-controlled, our analysis reveals a trade-off between short-term error reduction and long-term optimality. For completeness, we also discuss the squared loss case and outline appropriate update schemes, whose analysis requires additional care. Extensive simulation studies corroborate our theoretical findings.

Graph-based machine learning methods are valuable tools for identifying and predicting variation in genetic data. In particular, understanding phenotypic effects at the cellular level is an accelerating area in pharmacogenomics. Insight into how drugs or disease affect bio-networks could aid drug development and precision medicine. This article proposes a novel graph-theoretic approach to infer a co-occurrence network from 16S microbiome data, designed specifically for smallsample datasets. Such datasets pose challenges due to sparsity, compositionality, and complex interactions. The methodology includes steps to enrich and statistically filter the inferred networks. The approach extracts informative, feature-rich, biologically meaningful, and statistically significant networks from limited data. While tailored for small datasets, it is broadly applicable and can be extended to multi-omics integration. The method is tested on data from chickens vaccinated and challenged with Eimeria tenella. Genetic reads are processed, and networks inferred to characterize intestinal ecosystems at three disease progression stages. Analysis of network features yields biologically intuitive conclusions using statistical methods. Notably, the distribution of node features evolves with disease progression, and distributions reveal mutualistic and parasitic species clusters. A sub-network consistently appears across all conditions, suggesting a 'persistent microbiome'. A clustering algorithm is also applied to demonstrate the methods utility for downstream analysis.

We study the contextual bandits with knapsack (CBwK) problem under the high-dimensional setting where the dimension of the feature is large. We investigate how to exploit the sparsity structure to achieve improved regret for the CBwK problem. To this end, we first develop an online variant of the hard thresholding algorithm that performs the optimal sparse estimation. We further combine our online estimator with a primal-dual framework, where we assign a dual variable to each knapsack constraint and utilize an online learning algorithm to update the dual variable, thereby controlling the consumption of the knapsack capacity. We show that this integrated approach allows us to achieve a sublinear regret that depends logarithmically on the feature dimension, thus improving the polynomial dependency established in the previous literature. We also apply our framework to the high-dimension contextual bandit problem without the knapsack constraint and achieve optimal regret in both the data-poor regime and the data-rich regime.

Direction of arrival (DOA) can be estimated through sparse recovery (SR) methods based on the sparsity of signals. However, conventional SR-DOA methods, such as the alternating direction method of multipliers (ADMM), encounter issues such as difficulty in parameter setting and insufficient estimation accuracy. ADMM-Net is a neural network that combines ADMM with deep unfolding (DU). In this letter, we propose a method for estimating DOA using ADMM-Net. The simulation results demonstrate that the proposed method achieves a better balance between DOA estimation performance and real-time constraints.

The angular systematic error of multichannel airspace rotating radars can be precisely compensated, while the angular random error must be improved by increasing the signal-to-clutter-plus-noise ratio (SCNR). However, the echoes of the fast airspace rotating radars are subject to robust clutter effects, Doppler scattering and target energy scattering, which reduce the output SCNR. To solve this problem, an advanced ISTAP-based multichannel radar direction-of-arrival (DOA) estimation method is proposed. Through echo modeling and analysis of airborne multichannel radar with airspace rotation, a perturbation matrix is introduced to accurately characterize the rotation effect to improve clutter suppression. Experimental results show that the proposed method can compensate the unfavorable effects caused by fast airspace rotation, improve the SCNR, and maintain the DOA estimation accuracy.

MOTIVATION: Tuberculosis (TB) is caused by members of the Mycobacterium tuberculosis complex (MTBC), which has a strain- or lineage-based clonal population structure. The evolution of drug-resistance in the MTBC poses a threat to successful treatment and eradication of TB. Machine learning approaches are being increasingly adopted to predict drug-resistance and characterize underlying mutations from whole genome sequences. However, such approaches may not generalize well in clinical practice due to confounding from the population structure of the MTBC. RESULTS: To investigate how population structure affects machine learning prediction, we compared three different approaches to reduce lineage dependency in random forest (RF) models, including stratification, feature selection, and feature weighted models. All RF models achieved moderate-high performance (area under the ROC curve range: 0.60-0.98). First-line drugs had higher performance than second-line drugs, but it varied depending on the lineages in the training dataset. Lineage-specific models generally had higher sensitivity than global models which may be underpinned by strain-specific drug-resistance mutations or sampling effects. The application of feature weights and feature selection approaches reduced lineage dependency in the model and had comparable performance to unweighted RF models. AVAILABILITY AND IMPLEMENTATION: https://github.com/NinaMercedes/RF_lineages.

A large volume of commuting private cars cause serious traffic congestion, especially during morning and evening rush hours, and the low occupancy rate of commuting private cars bring a huge waste of resources. Carpooling among commuting private cars can reduce vehicle volume and alleviate traffic congestion. Moreover, commuters matching is the key issue to solve for realizing stable carpooling. This paper designs a commuting trajectory based stable carpooling model (CT-CSC) for commuting private cars, and proposes a particle swarm-based commuter matching approach for stable carpooling (PSCMA). The objective of CT-CSC model is minimizing carpooling cost. In the proposed PSCMA, the particle swarm and fitness calculation rules are redesigned to make it suitable for the CT-CSC model. The real-world RFID electronic identification data in Chongqing is used for experimental verification. Experimental results show the effect of the parameter inertia factor ω for the PSCMA. In addition, compared with genetic algorithm, hill climbing algorithm and simulation annealing algorithm, the performance of the PSCMA is outstanding. Furthermore, 1,003 commuters passing through the Huanghuayuan Bridge in Chongqing are selected to carpool, and we analyzed the reduction of the number of commuters, mileages and gasoline

Ride-sharing cars are private vehicles held by individuals or provided by ride-hailing companies for designated drivers to offer taxi-like services. Recently, various ride-sharing cars have emerged around the city with the popularity of online ride-hailing services. Identifying them is the critical task of transportation management. However, less work focuses on this issue due to the lack of city-wide private vehicles’ trajectory data and labeled ride-sharing cars. Fortunately, data collected by advanced sensing technology, such as electronic registration identification (ERI) of the motor vehicle data collected by radio-frequency identification (RFID) technology, provide us with an opportunity to detect ride-sharing cars from a data-driven aspect. This article proposes detecting ride-sharing cars via self-taught learning (STL) using ERI data, named STL-detector, which is accurate with very little labeled information. In detail, STL-detector consists of two components. In theunsupervised feature learningcomponent, we construct a 3-D convolutional neural networks (3-D-CNN) autoencoder trained with an amount of unlabeled data, which forms a succinct high-level input representation and significantly improve detection performance. In thesupervised classificationcomponent, we utilize the random forest (RF) as the classifier, which is trained on very little labeled data, to detect ride-sharing cars/others. The experimental results demonstrate that our STL-detector model can detect ride-sharing cars with better performance compared with other baselines on a set of train and test samples. Furthermore, we apply our model to a real-world scenario to detect ride-sharing cars and conduct a comparative analysis on the behavior of detected ride-sharing cars and taxis.

With the deepening of the urbanization process, the ownership of urban private cars continues to increase, resulting in severe urban traffic congestion and environmental problems. The customized bus, as an emerging public transportation mode, is considered an effective means to alleviate the above problems. This article employs electronic registration identification (ERI) data of vehicles to design customized buses for private cars, consisting of two components: 1) discovering private car commuters and 2) designing customized bus schemes. First, based on the spatial–temporal similarity and high-frequency characteristics of commuting trips, we mined the urban private car commuters and their corresponding commuting trips as the demand for customized buses. Then, we constructed the customized bus model, which targets the number of served passengers with the constraints, such as the trip time window, bus capacity, passenger load rate, etc. In the model, intermediate stops are not set to ensure bus punctuality and passenger experience, and buses of various capacities are employed to ensure effectiveness and efficiency. The differential evolution algorithm was utilized to find the optimal solution for the model. In the experiments, we carried out relevant verification based on Chongqing’s one-week ERI data. The experimental results showed customized bus schemes for various cases and verified the superior performance of our algorithm by comparing it with general optimization algorithms. Besides, through numerical calculation and traffic simulation, the excellent potential for customized buses in reducing urban transportation energy consumption and urban road congestion is illustrated.

The tensor train (TT) format enjoys appealing advantages in handling structural high-order tensors. The recent decade has witnessed the wide applications of TT-format tensors from diverse disciplines, among which tensor completion has drawn considerable attention. Numerous fast algorithms, including the Riemannian gradient descent (RGrad), have been proposed for the TT-format tensor completion. However, the theoretical guarantees of these algorithms are largely missing or sub-optimal, partly due to the complicated and recursive algebraic operations in TT-format decomposition. Moreover, existing results established for the tensors of other formats, for example, Tucker and CP, are inapplicable because the algorithms treating TT-format tensors are substantially different and more involved. In this paper, we provide, to our best knowledge, the first theoretical guarantees of the convergence of RGrad algorithm for TT-format tensor completion, under a nearly optimal sample size condition. The RGrad algorithm converges linearly with a constant contraction rate that is free of tensor condition number without the necessity of re-conditioning. We also propose a novel approach, referred to as the sequential second-order moment method, to attain a warm initialization under a similar sample size requirement. As a byproduct, our result even significantly refines the prior investigation of RGrad algorithm for matrix completion. Lastly, statistically (near) optimal rate is derived for RGrad algorithm if the observed entries consist of random sub-Gaussian noise. Numerical experiments confirm our theoretical discovery and showcase the computational speedup gained by the TT-format decomposition.

Private cars’ travel has been one of the main factors causing urban traffic congestion. Especially during morning and evening rush hours, private car commuters bring a significant burden to traffic. However, there is very little literature on them due to the lack of access to relevant data. A real-world dataset containing vehicle passing records of Electronic Registration Identification (ERI) of vehicles provides us with an opportunity to research private car commuters. We propose a regular behavior-based model to recognize private car commuters. In the model, a regular behavior-based definition of private car commuters is firstly proposed. Then, TDSP(Time dependent shortest path)-based distance measurement and a hierarchical clustering method are designed to extract regular behaviors. Furthermore, we utilize a regular threshold$p $to help determine regular behaviors. The experiment, which is conducted on a real-world dataset containing one-week vehicle passing records in Chongqing of China, validates the effectiveness and accuracy of the proposed model. Moreover, we analyze the mobility pattern of private car commuters, and some typical mobility patterns of them are successfully found.

Timestamps are often problematic in Internet-of-Things (IoT) systems due to time synchronization problems of distributed radio-frequency identification (RFID) readers or sensors. This issue may seriously affect the data quality in some fields, such as transportation. A typical IoT application in transportation is electronic registration identification of the motor vehicle (ERI), an emerging traffic data acquisition technology based on RFID. ERI data play a vital role in intelligent transportation. However, the data quality is often affected seriously by the inaccurate timestamps, which arise from the time-unsynchronized distributed ERI readers. To solve this issue, we propose a novel framework, data repairing of time synchronization problems (DR-TSP), which can detect the time-unsynchronized ERI readers and correct timestamp-deviated ERI data. Precisely, DR-TSP consists of three components. Problem reader discovery component employs a statistics-based method to detect the time-unsynchronized ERI readers and discovers the clock leaps of the problematic ERI reader through a smoothing-based method. Travel-time estimation component constructs a spatial correlative travel-time estimation model based on the neural network to infer timestamp deviation. The influence of clock deviation is considered in the model training. Data correction component utilizes the above results to correct the timestamp-deviated data. Experiments over large-scale ERI data collected from a big China city, Chongqing, show that our method can significantly improve data quality.

In this article, we investigate effective sketching schemes via sparsification for high dimensional multilinear arrays or tensors. More specifically, we propose a novel tensor sparsification algorithm that retains a subset of the entries of a tensor in a judicious way, and prove that it can attain a given level of approximation accuracy in terms of tensor spectral norm with a much smaller sample complexity when compared with existing approaches. In particular, we show that for akth order$ {d}\times \cdots \times {d}$cubic tensor ofstable rank$ {r}_{ {s}}$, the sample size requirement for achieving a relative error$\varepsilon $is, up to a logarithmic factor, of the order$ {r}_{ {s}}^{1/2} {d}^{ {k}/2} /\varepsilon $when$\varepsilon $is relatively large, and$ {r}_{ {s}} {d} /\varepsilon ^{2}$and essentially optimal when$\varepsilon $is sufficiently small. It is especially noteworthy that the sample size requirement for achieving a high accuracy is of an order independent ofk. To further demonstrate the utility of our techniques, we also study how higher order singular value decomposition (HOSVD) of large tensors can be efficiently approximated via sparsification.

Urban mobility is enjoying much attention due to increasingly serious traffic and environment problems in cities. Private cars are the most important component of urban road traffic. However, current research on urban mobility seldom employs travel data from private cars due to the lack of access to corresponding data acquisition. This problem can be solved with the massive application of Electronic Registration Identification (ERI), which is an emerging technology to identify a unique vehicle based on Radio Frequency Identification (RFID). This paper proposes a framework for discovering the urban mobility of private cars based on ERI data. The main research content includes two parts: trajectory segmentation and attractive area mining. In the trajectory segmentation, stay segments in trajectories are identified by Bayes classification based on the link travel time distribution model. The model parameters of each link are trained by Expectation Maximization(EM) algorithm. In attractive area mining, a spatial clustering algorithm based on data field is introduced. Finally, we utilized real-world data into the proposed algorithms. The experimental results show that the proposed method can accurately segment the trajectory, and the visualization of attractive areas reveals the urban mobility characteristics of private cars.

The higher order singular value decomposition (HOSVD) of tensors is a generalization of matrix SVD. The perturbation analysis of HOSVD under random noise is more delicate than its matrix counterpart. Recently, polynomial time algorithms have been proposed where statistically optimal estimates of the singular subspaces and the low rank tensors are attainable in the Euclidean norm. In this article, we analyze the sup-norm perturbation bounds of HOSVD and introduce estimators of the singular subspaces with sharp deviation bounds in the sup-norm. We also investigate a low rank tensor denoising estimator and demonstrate its fast convergence rate with respect to the entry-wise errors. The sup-norm perturbation bounds reveal unconventional phase transitions for statistical learning applications such as the exact clustering in high dimensional Gaussian mixture model and the exact support recovery in sub-tensor localizations. In addition, the bounds established for HOSVD also elaborate the one-sided sup-norm perturbation bounds for the singular subspaces of unbalanced (or fat) matrices.

Low-rank matrix regression refers to the instances of recovering a low-rank matrix based on specially designed measurements and the corresponding noisy outcomes. Numerous statistical methods have been developed over the recent decade for efficiently reconstructing the unknown low-rank matrices. It is often interesting, in certain applications, to estimate the unknown singular subspaces. In this paper, we revisit the low-rank matrix regression model and introduce a two-step procedure to construct confidence regions of the singular subspaces. We investigate distributions of the joint projection distance between the empirical singular subspaces and the unknown true singular subspaces. We prove asymptotical normality of the joint projection distance with data-dependent centering and normalization when r3/2(m1+ m2)3/2= o(n/log n) where m1, m2denote the matrix row and column sizes, r is the rank and n is the number of independent random measurements. Consequently, data-dependent confidence regions of the true singular subspaces are established which attain pre-determined confidence levels asymptotically. Additionally, non-asymptotic convergence rates are also established. Numerical results are presented to show the merits of our methods.

In this paper, we propose a general framework for tensor singular value decomposition (tensor singular value decomposition (SVD)), which focuses on the methodology and theory for extracting the hidden low-rank structure from high-dimensional tensor data. Comprehensive results are developed on both the statistical and computational limits for tensor SVD. This problem exhibits three different phases according to the signal-to-noise ratio (SNR). In particular, with strong SNR, we show that the classical higher-order orthogonal iteration achieves the minimax optimal rate of convergence in estimation; with weak SNR, the information-theoretical lower bound implies that it is impossible to have consistent estimation in general; with moderate SNR, we show that the non-convex maximum likelihood estimation provides optimal solution, but with NP-hard computational cost; moreover, under the hardness hypothesis of hypergraphic planted clique detection, there are no polynomial-time algorithms performing consistently in general.

Rate adaptation varies the transmission rate of a wireless sender to match the wireless channel conditions, in order to achieve the best possible performance. It is a key component of IEEE 802.11 wireless networks. Minstrel is a popular rate adaptation algorithm due to its efficiency and availability in commonly used wireless drivers. However, despite its popularity, little work has been done on evaluating the performance of Minstrel or comparing it to the performance of fixed rates. In this paper, we conduct an experimental study that compares the performance of Minstrel against fixed rates in an IEEE 802.11g testbed. The experiment results show that whilst Minstrel performs reasonably well in static wireless channel conditions, in some cases the algorithm has difficulty selecting the optimal data rate in the presence of dynamic channel conditions. In addition, Minstrel performs well when the channel condition improves from bad quality to good quality. However, Minstrel has trouble selecting the optimal rate when the channel condition deteriorates from good quality to bad quality.

In this paper, a novel concept called a uniquely factorable constellation pair (UFCP) is proposed for the systematic design of a noncoherent full diversity collaborative unitary space-time block code by normalizing two Alamouti codes for a wireless communication system having two transmitter antennas and a single receiver antenna. It is proved that such a unitary UFCP code assures the unique identification of both channel coefficients and transmitted signals in a noise-free case as well as full diversity for the noncoherent maximum likelihood receiver in a noise case. To further improve error performance, an optimal unitary UFCP code is designed by appropriately and uniquely factorizing a pair of energy-efficient cross quadrature amplitude modulation (QAM) constellations to maximize the coding gain subject to a transmission bit rate constraint. After a deep investigation of the fractional coding gain function, a technical approach developed in this paper to maximizing the coding gain is to carefully design an energy scale to compress the first three largest energy points in the corner of the QAM constellations in the denominator of the objective as well as carefully design a constellation triple forming two UFCPs, with one collaborating with the other two so as to make the accumulated minimum Euclidean distance along the two transmitter antennas in the numerator of the objective as large as possible, and at the same time, to avoid as many corner points of the QAM constellations with the largest energy as possible to achieve the minimum of the numerator. In other words, the optimal coding gain is attained by intelligent constellations collaboration and efficient energy compression. Computer simulations demonstrate that error performance of the optimal unitary UFCP code presented in this paper outperforms those of the differential code and the signal-to-noise-ratio-efficient training code.

IEEE 802.11 wireless systems usually contain rate control algorithms which are designed to adapt the transmission rate between several available rates in response to varying channel conditions. The efficiency of the rate control algorithm in selecting optimal data rates for the channel conditions directly impacts on the throughput of the wireless system. Minstrel is a rate control algorithm that has good performance compared with other algorithms, and is widely implemented in popular wireless drivers such as MadWiFi, Ath5k and Ath9k. However, surprisingly, there is very little literature studying the performance of Minstrel. In this paper, we present an experimental study using a commercial wireless access point which evaluates the performance of Minstrel against fixed rates in a real-world IEEE 802.11g testbed. This experimental study takes a number of factors into account, such as uplink and downlink channels, transmission power, channel dynamics, Access Point (AP) location, and datagram size. The results show that while Minstrel performs well in many cases (particularly with “good” or “stable” channel conditions), the algorithm has difficulty achieving optimal throughput performance with “poor” or highly “dynamic” channel conditions. Our findings provide useful information on the design of rate control algorithms.