K-plane clustering (KPC), hyperplane clustering, and mixture regression all essentially fall within the same class of problems. This problem can be conceptualized as clustering in relatively high-dimensional K subspaces or K linear manifolds. Traditional KPC or fuzzy KPC models demonstrate a pronounced susceptibility to outliers, as they presuppose that the projection distance between data points and the plane normal vector adheres to the$L_{2}$distance. Meanwhile, the assumption of infinitely extending clusters adversely affects clustering performance. To solve these problems, this paper proposed a new robust fuzzy local k-plane clustering (RFLkPC) method that combines the mixture distance of hinge loss and$L_{1}$norm. The RFLkPC model assumes that each plane cluster is bounded to a finite area, which can flexibly and robustly handle plane clustering tasks with outliers or not. The corresponding model and optimization algorithms of RFLkPC were provided. Compared to other related models on this topic, a large number of experiments verify the efficiency of RFLkPC on simulated data and real data.

Deep reinforcement learning (RL) has shown remarkable success in specific offline decision-making scenarios, yet its theoretical guarantees are still under development. Existing works on offline RL theory primarily emphasize a few trivial settings, such as linear MDP or general function approximation with strong assumptions and independent data, which lack guidance for practical use. The coupling of deep learning and Bellman residuals makes this problem challenging, in addition to the difficulty of data dependence. In this paper, we establish a non-asymptotic estimation error of pessimistic offline RL using general neural network approximation with C-mixing data regarding the structure of networks, the dimension of datasets, and the concentrability of data coverage, under mild assumptions. Our result shows that the estimation error consists of two parts: the first converges to zero at a desired rate on the sample size with partially controllable concentrability, and the second becomes negligible if the residual constraint is tight. This result demonstrates the explicit efficiency of deep adversarial offline RL frameworks. We utilize the empirical process tool for C-mixing sequences and the neural network approximation theory for the Holder class to achieve this. We also develop methods to bound the Bellman estimation error caused by function approximation with empirical Bellman constraint perturbations. Additionally, we present a result that lessens the curse of dimensionality using data with low intrinsic dimensionality and function classes with low complexity. Our estimation provides valuable insights into the development of deep offline RL and guidance for algorithm model design.

Recent research shows that pre-trained language models (PLMs) suffer from “prompt bias” in factual knowledge extraction, i.e., prompts tend to introduce biases toward specific labels. Prompt bias presents a significant challenge in assessing the factual knowledge within PLMs. Therefore, this paper aims to improve the reliability of existing benchmarks by thoroughly investigating and mitigating prompt bias. We show that: 1) all prompts in the experiments exhibit non-negligible bias, with gradient-based prompts like AutoPrompt and OptiPrompt displaying significantly higher levels of bias; 2) prompt bias can amplify benchmark accuracy unreasonably by overfitting the test datasets, especially on imbalanced datasets like LAMA. Based on these findings, we propose a representation-based approach to mitigate the prompt bias during inference time. Specifically, we first estimate the biased representation using prompt-only querying, and then remove it from the model’s internal representations to generate the debiased representations, which are used to produce the final debiased outputs. Experiments across various prompts, PLMs, and benchmarks show that our approach can not only correct the overfitted performance caused by prompt bias, but also significantly improve the prompt retrieval capability (up to 10% absolute performance gain). These results indicate that our approach effectively alleviates prompt bias in knowledge evaluation, thereby enhancing the reliability of benchmark assessments. Hopefully, our plug-and-play approach can be a golden standard to strengthen PLMs toward reliable knowledge bases. Code and data are released in https://github.com/FelliYang/PromptBias.

Understanding the power of parameterized quantum circuits (PQCs) in accomplishing machine learning tasks is one of the most important questions in quantum machine learning. In this paper, we focus on the PQC expressivity for general multivariate function classes. Previously established Universal Approximation Theorems for PQCs are either nonconstructive or assisted with parameterized classical data processing, making it hard to justify whether the expressive power comes from the classical or quantum parts. We explicitly construct data re-uploading PQCs for approximating multivariate polynomials and smooth functions and establish the first non-asymptotic approximation error bounds for such functions in terms of the number of qubits, the quantum circuit depth and the number of trainable parameters of the PQCs. Notably, we show that for multivariate polynomials and multivariate smooth functions, the quantum circuit size and the number of trainable parameters of our proposed PQCs can be smaller than the deep ReLU neural networks. We further demonstrate the approximation capability of PQCs via numerical experiments. Our results pave the way for designing practical PQCs that can be implemented on near-term quantum devices with limited resources.

The membership matrix is a key element in fuzzy clustering, enabling novel data representation in multiple clusters. The row vectors of the membership matrix represent each sample's degree of membership to different clusters. Notably, researchers have confirmed the presence of the local affinity among these row vectors, effectively preserving the local structure of the original data distribution. However, in this work, we consider that most sample points have insignificant fuzziness, with fuzziness found primarily in a few clusters, resulting in most membership vectors being sparse. To tackle this issue, we present the sparse, membership-affinity fuzzy clustering model, which leverages the sparsity of the row vectors and its affinity to establish a more appropriate representation, along with an optimization algorithm. Our experimental results on both simulated and real datasets demonstrate that the combination of sparsity and affinity can significantly enhance fuzzy clustering performance over other models.

Based on the offset Rademacher complexity, this work outlines a systematical framework for deriving sharp excess risk bounds in statistical learning without Bernstein condition. In addition to recovering fast rates in a unified way for some parametric and nonparametric supervised learning models with minimum identifiability assumptions, we also obtain new and improved results for LAD (sparse) linear regression and deep logistic regression with deep ReLU neural networks, respectively.

Improving the generalization of neural networks under domain shift is an important and challenging task in computer vision. Obtaining an invariant representation across domains is a benchmark method in the literature. In this paper, we propose an invariant and sufficient supervised representation learning (ISSRL) approach to learn a domain invariant representation which is also preserving information used for downstream tasks. To this end, we formulate ISSRL by finding a nonlinear map$\boldsymbol{g}$such that$Y\perp X\vert \boldsymbol{g}(X)$and$(Y,\boldsymbol{g}(X))\perp D$at the population level, where D is the label of the domains and$(X, Y)$is the paired data sampled from domains with label. We use distance correlation to characterize the (conditional) independence. At the sample level, we construct a novel loss function through an unbiased empirical version of distance correlation. We train the representation map by parameterizing it with deep neural networks. Both simulation study and real data evaluation show that ISSRL outperforms the state-of-the-art on out-of-distribution generalization. The PyTorch code for ISSRL is available at https://github.com/CaC033/ISSRL.

In this paper, we propose and analyze a reconstruction algorithm for imaging an anisotropic conductivity tensor in a second-order elliptic PDE with a nonzero Dirichlet boundary condition from internal current densities. It is based on a regularized output least-squares formulation with the standard $L^2(\Omega)^{d,d}$ penalty, which is then discretized by the standard Galerkin finite element method. We establish the continuity and differentiability of the forward map with respect to the conductivity tensor in the $L^p(\Omega)^{d,d}$-norms, the existence of minimizers and optimality systems of the regularized formulation using the concept of H-convergence. Further, we provide a detailed analysis of the discretized problem, especially the convergence of the discrete approximations with respect to the mesh size, using the discrete counterpart of H-convergence. In addition, we develop a projected Newton algorithm for solving the first-order optimality system. We present extensive two-dimensional numerical examples to show the efficiency of the proposed method.

The normalized difference vegetation index (NDVI) can reflect the plant life cycle of growth and senescence and has become a widely used tool for many applications related to phenology, ecology, and environment. However, unwanted disturbance from cloud, snow, and other atmospheric effects greatly lowers the NDVI quality and hinders its further application. In this article, differing from the previous research attempting to approach the upper NDVI envelope by local adjustment or threshold-related iteration, a novel one-step global variational reconstruction (OGVR) method for NDVI time series is proposed via joint modeling of the gradual vegetation change and negatively biased atmospheric contamination. Two versions of the proposed method are designed for processing NDVI data with or without auxiliary flag information. Long-term and global-scale Advanced Very High Resolution Radiometer (AVHRR) global inventory monitoring and modeling system (GIMMS) data were applied in simulated and real-data experiments to verify the proposed method. The results show that the proposed method can successfully estimate the natural vegetation change from seriously contaminated NDVI time series and can conquer the problem of continuous low-value gaps. The qualitative and quantitative comparisons with five other widely used methods indicate that the proposed method has significant advantages in terms of both effectiveness and stability.

Fuzzy clustering generates a membership vector for each data point in the dataset to indicate its belongingness to different clusters. This procedure can be regarded as an encoding process and the obtained vectors of memberships are the new representations of original data. Naturally, the affinities between new representations or the vectors of memberships should be consistent with the ones between original data points. For example, the data points close to each other should also take similar membership vectors. Such constraints on the affinities of memberships are valuable prior knowledge that should be imposed to the objective function of fuzzy clustering for better performance. To this end, we introduce the membership affinity lasso for fuzzy clustering in this paper. Utilizing alternating direction method of multipliers, an efficient approach is derived to optimize the general membership affinity lasso regularized fuzzy clustering model in offline manner. As illustrative examples, three new fuzzy clustering algorithms with the membership affinity lasso are proposed. Experiments on the synthetic and real data demonstrate the superiority and flexibility of the proposed algorithms.

We propose a constructive approach to estimating sparse, high-dimensional linear regression models. The approach is a computational algorithm motivated from the KKT conditions for the $\ell_0$-penalized least squares solutions. It generates a sequence of solutions iteratively, based on support detection using primal and dual information and root finding. We refer to the algorithm as SDAR for brevity. Under a sparse Riesz condition on the design matrix and certain other conditions, we show that with high probability, the $\ell_2$ estimation error of the solution sequence decays exponentially to the minimax error bound in $O(\log(R\sqrt{J}))$ iterations, where $J$ is the number of important predictors and $R$ is the relative magnitude of the nonzero target coefficients; and under a mutual coherence condition and certain other conditions, the $\ell_{\infty}$ estimation error decays to the optimal error bound in $O(\log(R))$ iterations. Moreover the SDAR solution recovers the oracle least squares estimator within a finite number of iterations with high probability if the sparsity level is known. Computational complexity analysis shows that the cost of SDAR is $O(np)$ per iteration. We also consider an adaptive version of SDAR for use in practical applications where the true sparsity level is unknown. Simulation studies demonstrate that SDAR outperforms Lasso, MCP and two greedy methods in accuracy and efficiency.

Striping effects are a common phenomenon in remote-sensing imaging systems, and they can exhibit considerable differences between different sensors. Such artifacts can greatly degrade the quality of the measured data and further limit the subsequent applications in higher level remote-sensing products. Although a lot of destriping methods have been proposed to date, a few of them are robust to different types of stripes. In this paper, we conduct a thorough feature analysis of stripe noise from a novel perspective. With regard to the problem of striping diversity and complexity, we propose a universal destriping framework. In the proposed destriping procedure, a 1-D variational method is first designed and utilized to estimate the statistical feature-based guidance. The guidance information is then incorporated into 2-D optimization to control the image estimation for a reliable and clean output. The iteratively reweighted least-squares method and alternating direction method of multipliers are exploited in the proposed approach to solve the minimization problems. Experiments under various cases of simulated and real stripes confirm the effectiveness and robustness of the proposed model in terms of the qualitative and quantitative comparisons with other approaches.

In this note, we analyze an iterative soft/hard thresholding algorithm with homotopy continuation for recovering a sparse signal x†from noisy data of a noise level ε. Under suitable regularity and sparsity conditions, we design a path, along which the algorithm can find a solution x*, which admits a sharp reconstruction error ||x* - x†|| ℓ∞ = O(ε) with an iteration complexity O((ln ε)/(ln γ)np), where n and p are problem dimensionality and γ ε (0,1) controls the length of the path. Numerical examples are given to illustrate its performance.

Remote sensing images are often contaminated by varying degrees of stripes, which severely affects the visual quality and subsequent application of the data. Unlike with conventional methods, we achieve the destriping by separating the stripe component based on a full analysis of the various stripe properties. Under an optimization framework, an ℓ0-norm-based regularization is used to characterize the global sparse distribution of the stripes. In addition, difference-based constraints are adopted to describe the local smoothness and discontinuity in the along-stripe and across-stripe directions, respectively. The alternating direction method of multipliers is applied to solve and accelerate the model optimization. Experiments with both simulated and real data demonstrate the effectiveness of the proposed model, in terms of both qualitative and quantitative perspectives.