Time series generation is widely used in real-world applications such as simulation, data augmentation, and hypothesis testing. Recently, diffusion models have emerged as the de facto approach to time series generation, enabling diverse synthesis scenarios. However, the fixed standard-Gaussian diffusion prior may be ill-suited for time series data, which exhibit properties such as temporal order and fixed time points. In this paper, we propose TimeBridge, a framework that flexibly synthesizes time series data by using diffusion bridges to learn paths between a chosen prior and the data distribution. We then explore several prior designs tailored to time series synthesis. Our framework covers (i) data- and time-dependent priors for unconditional generation and (ii) scale-preserving priors for conditional generation. Experiments show that our framework with data-driven priors outperforms standard diffusion models on time series generation.

With the increasing need to safeguard data privacy in machine learning models, differential privacy (DP) is one of the major frameworks to build privacy-preserving models. Support Vector Machines (SVMs) are widely used traditional machine learning models due to their robust margin guarantees and strong empirical performance in binary classification. However, applying DP to multi-class SVMs is inadequate, as the standard one-versus-rest (OvR) and one-versus-one (OvO) approaches repeatedly query each data sample when building multiple binary classifiers, thus consuming the privacy budget proportionally to the number of classes. To overcome this limitation, we explore all-in-one SVM approaches for DP, which access each data sample only once to construct multi-class SVM boundaries with margin maximization properties. We propose a novel differentially Private Multi-class SVM (PMSVM) with weight and gradient perturbation methods, providing rigorous sensitivity and convergence analyses to ensure DP in all-in-one SVMs. Empirical results demonstrate that our approach surpasses existing DP-SVM methods in multi-class scenarios.

Diffusion models have shown their effectiveness in generation tasks by well-approximating the underlying probability distribution. However, diffusion models are known to suffer from an amplified inherent bias from the training data in terms of fairness. While the sampling process of diffusion models can be controlled by conditional guidance, previous works have attempted to find empirical guidance to achieve quantitative fairness. To address this limitation, we propose a fairness-aware sampling method called \textit{attribute switching} mechanism for diffusion models. Without additional training, the proposed sampling can obfuscate sensitive attributes in generated data without relying on classifiers. We mathematically prove and experimentally demonstrate the effectiveness of the proposed method on two key aspects: (i) the generation of fair data and (ii) the preservation of the utility of the generated data.

To alleviate the utility degradation of deep learning image classification with differential privacy (DP), employing extra public data or pre-trained models has been widely explored. Recently, the use of in-distribution public data has been investigated, where tiny subsets of datasets are released publicly. In this paper, we investigate a framework that leverages recent diffusion models to amplify the information of public data. Subsequently, we identify data diversity and generalization gap between public and private data as critical factors addressing the limited public data. While assuming 4% of training data as public, our method achieves 85.48% on CIFAR-10 with a privacy budget of$\varepsilon, =2$, without employing extra public data for training.

Time series forecasting is crucial for applications across multiple domains and various scenarios. Although Transformers have dramatically advanced the landscape of forecasting, their effectiveness remains debated. Recent findings have indicated that simpler linear models might outperform complex Transformer-based approaches, highlighting the potential for more streamlined architectures. In this paper, we shift the focus from evaluating the overall Transformer architecture to specifically examining the effectiveness of self-attention for time series forecasting. To this end, we introduce a new architecture, Cross-Attention-only Time Series transformer (CATS), that rethinks the traditional transformer framework by eliminating self-attention and leveraging cross-attention mechanisms instead. By establishing future horizon-dependent parameters as queries and enhanced parameter sharing, our model not only improves long-term forecasting accuracy but also reduces the number of parameters and memory usage. Extensive experiment across various datasets demonstrates that our model achieves superior performance with the lowest mean squared error and uses fewer parameters compared to existing models. The implementation of our model is available at: https://github.com/dongbeank/CATS.

Training deep learning models with differential privacy (DP) results in a degradation of performance. The training dynamics of models with DP show a significant difference from standard training, whereas understanding the geometric properties of private learning remains largely unexplored. In this paper, we investigate sharpness, a key factor in achieving better generalization, in private learning. We show that flat minima can help reduce the negative effects of per-example gradient clipping and the addition of Gaussian noise. We then verify the effectiveness of Sharpness-Aware Minimization (SAM) for seeking flat minima in private learning. However, we also discover that SAM is detrimental to the privacy budget and computational time due to its two-step optimization. Thus, we propose a new sharpness-aware training method that mitigates the privacy-optimization trade-off. Our experimental results demonstrate that the proposed method improves the performance of deep learning models with DP from both scratch and fine-tuning. Code is available at https://github.com/jinseongP/DPSAT.

The success of deep learning is greatly attributed to stochastic gradient descent (SGD), yet it remains unclear how SGD finds well-generalized models. We demonstrate that SGD has an implicit regularization effect on the logit-weight Jacobian norm of neural networks. This regularization effect is weighted with the *impurity* of the probability output, and thus it is active in a certain phase of training. Moreover, based on these findings, we propose a novel optimization method that explicitly regularizes the Jacobian norm, which leads to similar performance as other state-of-the-art sharpness-aware optimization methods.

This study presents the first differentially private and fair clustering method, built on the recently proposed density-based fair clustering approach. The method addresses the limitations of fair clustering algorithms that necessitate the use of sensitive personal information during training or inference phases. Two novel solutions, the Gaussian mixture density function and Voronoi cell, are proposed to enhance the method's performance in terms of privacy, fairness, and utility compared to previous methods. The experimental results on both synthetic and real-world data confirm the compatibility of the proposed method with differential privacy, achieving a better fairness-utility trade-off than existing methods when privacy is not considered. Moreover, the proposed method requires significantly less computation time, being at least 3.7 times faster than the state-of-the-art.

Adversarial training has become the de-facto standard method for improving the robustness of models against adversarial examples. However, robust overfitting remains a significant challenge, leading to a large gap between the robustness on the training and test datasets. To understand and improve robust generalization, various measures have been developed, including margin, smoothness, and flatness-based measures. In this study, we present a large-scale analysis of robust generalization to empirically verify whether the relationship between these measures and robust generalization remains valid in diverse settings. We demonstrate when and how these measures effectively capture the robust generalization gap by comparing over 1,300 models trained on CIFAR-10 under the $L_\infty$ norm and further validate our findings through an evaluation of more than 100 models from RobustBench across CIFAR-10, CIFAR-100, and ImageNet. We hope this work can help the community better understand adversarial robustness and motivate the development of more robust defense methods against adversarial attacks.

Deep learning is vulnerable to adversarial examples. Many defenses based on randomized neural networks have been proposed to solve the problem, but fail to achieve robustness against attacks using proxy gradients such as the Expectation over Transformation (EOT) attack. We investigate the effect of the adversarial attacks using proxy gradients on randomized neural networks and demonstrate that it highly relies on the directional distribution of the loss gradients of the randomized neural network. We show in particular that proxy gradients are less effective when the gradients are more scattered. To this end, we propose Gradient Diversity (GradDiv) regularizations that minimize the concentration of the gradients to build a robust randomized neural network. Our experiments on MNIST, CIFAR10, and STL10 show that our proposed GradDiv regularizations improve the adversarial robustness of randomized neural networks against a variety of state-of-the-art attack methods. Moreover, our method efficiently reduces the transferability among sample models of randomized neural networks.

Although fast adversarial training has demonstrated both robustness and efficiency, the problem of "catastrophic overfitting" has been observed. This is a phenomenon in which, during single-step adversarial training, the robust accuracy against projected gradient descent (PGD) suddenly decreases to 0% after a few epochs, whereas the robust accuracy against fast gradient sign method (FGSM) increases to 100%. In this paper, we demonstrate that catastrophic overfitting is very closely related to the characteristic of single-step adversarial training which uses only adversarial examples with the maximum perturbation, and not all adversarial examples in the adversarial direction, which leads to decision boundary distortion and a highly curved loss surface. Based on this observation, we propose a simple method that not only prevents catastrophic overfitting, but also overrides the belief that it is difficult to prevent multi-step adversarial attacks with single-step adversarial training.

Privacy in machine learning has been widely recognized as an essential ethical and legal issue, because the data used for machine learning may contain sensitive information. Homomorphic encryption has recently attracted attention as a key solution to preserve privacy in machine learning applications. However, current approaches on the training of encrypted machine learning have relied heavily on hyperparameter selection, which should be avoided owing to the extreme difficulty of conducting validation on encrypted data. In this study, we propose an effective privacy-preserving logistic regression method that is free from the approximation of the sigmoid function and hyperparameter selection. In our framework, a logistic regression model can be transformed into the corresponding ridge regression for the logit function. We provide a theoretical background for our framework by suggesting a new generalization error bound on the encrypted data. Experiments on various real-world data show that our framework achieves better classification results while reducing latency by $\sim68\%$, compared to the previous models.

We study the problem of training certifiably robust models against adversarial examples. Certifiable training minimizes an upper bound on the worst-case loss over the allowed perturbation, and thus the tightness of the upper bound is an important factor in building certifiably robust models. However, many studies have shown that Interval Bound Propagation (IBP) training uses much looser bounds but outperforms other models that use tighter bounds. We identify another key factor that influences the performance of certifiable training: \textit{smoothness of the loss landscape}. We find significant differences in the loss landscapes across many linear relaxation-based methods, and that the current state-of-the-arts method often has a landscape with favorable optimization properties. Moreover, to test the claim, we design a new certifiable training method with the desired properties. With the tightness and the smoothness, the proposed method achieves a decent performance under a wide range of perturbations, while others with only one of the two factors can perform well only for a specific range of perturbations. Our code is available at \url{https://github.com/sungyoon-lee/LossLandscapeMatters}.

In this study, we give a stability analysis of denoising autoencoder(DAE) from the novel perspective of dynamical systems when the input density is defined as a distribution on a manifold. We demonstrate the connection between the corrupted distribution and the learned reconstruction function of a nonlinear DAE, which motivates the use of a dynamic projection system (DPS) associated with the learned reconstruction function. Utilizing the constructed DPS, we prove that the high-density region of the corrupted data distribution asymptotically converges to the data manifold. Then, we show that the region is the attracting stable equilibrium manifold of the DPS which is completely stable. These results serve a theoretical basis of the DAE in recognizing the high-density region of the highly corrupted data with large deviations through the DPS. The effectiveness of this analysis is verified by conducting experiments on several toy examples and real image datasets with various types of noise.

Verifiable training is a promising research direction for training a robust network. However, most verifiable training methods are slow or lack scalability. In this study, we propose a fast and scalable certifiable training algorithm based on Lipschitz analysis and interval arithmetic. Our certifiable training algorithm provides a tight propagated outer bound by introducing the box constraint propagation (BCP), and it efficiently computes the worst logit over the outer bound. In the experiments, we show that BCP achieves a tighter outer bound than the global Lipschitz-based outer bound. Moreover, our certifiable training algorithm is over 12 times faster than the state-of-the-art dual relaxation-based method; however, it achieves comparable or better verification performance, improving natural accuracy. Our fast certifiable training algorithm with the tight outer bound can scale to Tiny ImageNet with verification accuracy of 20.1\% ($\ell_2$-perturbation of $\epsilon=36/255$). Our code is available at \url{https://github.com/sungyoon-lee/bcp}.

Natural language processing (NLP) is an important application area in domain adaptation because properties of texts depend on their corpus. However, a textual input is not fundamentally represented as the numerical vector. Many domain adaptation methods for NLP have been developed on the basis of n umerical representations of texts instead of textual inputs. Thus, we develop a distributed representation learning method of words and documents for domain adaptation. The developed method addresses the domain separation problem of document embeddings from different domains, that is, the supports of the embeddings are separable across domains and the distributions of the embeddings are discriminated. We propose a new method based on negative sampling. The proposed method learns document embeddings by assuming that a noise distribution is dependent on a domain. The proposed method moves a document embedding close to the embeddings of the important words in the document and keeps the embedding away from the word embeddings that occur frequently in both domains. For Amazon reviews, we verified that the proposed method outperformed other representation methods in terms of indiscriminability of the distributions of the document embeddings through experiments such as visualizing them and calculating a proxy A-distance measure. We also performed sentiment classification tasks to validate the effectiveness of document embeddings. The proposed method achieved consistently better results than other methods. In addition, we applied the learned document embeddings to the domain adversarial neural network method, which is a popular deep learning-based domain adaptation model. The proposed method obtained not only better performance on most datasets but also more stable convergences for all datasets than the other methods. Therefore, the proposed method are applicable to other domain adaptation methods for NLP using numerical representations of documents or words.

We present a compact modeling of human lung anatomy (with lung lobes, bronchial trees, and pulmonary blood vessels), and demonstrate a plausible deformation (with collision detection and avoidance) for the whole anatomical structure. For this purpose, we employ a hybrid hierarchy of various bounding volumes (generated by moving spheres). Separation lists and parallel processing are also used for the acceleration of collision detection and avoidance, all implemented in CPUs only. Experimental results show that our modeling scheme compresses conventional lung models (often commercially available as meshes) by around 70 times lighter, and the resulting human lung deformation supports an interactive-speed performance, with frame rate of 10--20 fps.

Support-based clustering using kernels suffers from serious computational limitations inherent in many kernel methods when applied to very large-scale problems despite its ability to identify clusters with complex shapes. In this paper, we propose a novel clustering algorithm called Voronoi cell-based clustering to expedite support-based clustering using kernels. In contrast to previous studies, including the basin cell-based method, the proposed method achieves computational efficiency in both the training phase to construct a support estimate using sampled data to reduce the evaluation of kernels and the labeling phase to assign a cluster label on each data point nearest its representative point. The performance superiority of the proposed method over the other basin cell-based methods in terms of computational time and storage efficiency is verified by various experiments using benchmark sets and in real applications to image segmentation.

We propose a transductive Gaussian process (TGP) regression method with regularized Laplacian kernels. Transductive learning exploits not only the labeled data but also the unlabeled test instances for learning. GPs are Bayesian probabilistic regressors which use only labeled data. To use unlabeled data in GPs, regularized Laplacian kernels are used. Similar to the case of a supervised GP regression, the proposed method provides not only the predicted target values but also their error bars. It also provides a hyperparameter selection method based on a Bayesian model selection scheme. We applied the proposed TGP method to the object pose estimation data sets as well as artificial data sets and compared the existing methods. Experimental results show that the proposed method has some advantages over the existing methods.

The search for a low-dimensional structure in high-dimensional data is one of the fundamental tasks in machine learning and pattern recognition. Manifold learning algorithms have recently emerged as alternatives to traditional linear dimension reduction techniques. In this paper, we propose a novel projection method that can be combined with any manifold learning methods to improve their dimension reduction performance when applied to high-dimensional data with a high level of noise. The method first builds a dispersion function that describes the distribution of dispersed manifold where the data lie. It then projects the noisy data onto a region wrapping the true manifold sufficiently close to it by applying a dynamical projection system associated with the constructed dispersion function. The effectiveness of the proposed projection method is validated by applying it to some real-world data sets with promising results.

Clustering methods utilizing support estimates of a data distribution have recently attracted much attention because of their ability to generate cluster boundaries of arbitrary shape and to deal with outliers efficiently. In this paper, we propose a novel dissimilarity measure based on a dynamical system associated with support estimating functions. Theoretical foundations of the proposed measure are developed and applied to construct a clustering method that can effectively partition the whole data space. Simulation results demonstrate that clustering based on the proposed dissimilarity measure is robust to the choice of kernel parameters and able to control the number of clusters efficiently.

In this brief, a novel method that constructs a sparse kernel machine is proposed. The proposed method generates attractors as sparse solutions from a built-in kernel machine via a dynamical system framework. By readjusting the corresponding coefficients and bias terms, a sparse kernel machine that approximates a conventional kernel machine is constructed. The simulation results show that the constructed sparse kernel machine improves the efficiency of testing phase while maintaining comparable test error.

We deal with the nonlinear manifold learning problem to find a low-dimensional structure in high-dimensional data. Based on Gaussian random fields framework, we propose an approximate sampling method for coordinates on the manifolds. Experimentally the mean of samples are shown to be almost equal to the coordinates obtained by locally linear embedding where the generated set of samples of coordinates show interesting variety.

In this letter, we develop a gaussian process model for clustering. The variances of predictive values in gaussian processes learned from a training data are shown to comprise an estimate of the support of a probability density function. The constructed variance function is then applied to construct a set of contours that enclose the data points, which correspond to cluster boundaries. To perform clustering tasks of the data points, an associated dynamical system is built, and its topological invariant property is investigated. The experimental results show that the proposed method works successfully for clustering problems with arbitrary shapes.

A novel learning algorithm for semisupervised classification is proposed. The proposed method first constructs a support function that estimates a support of a data distribution using both labeled and unlabeled data. Then, it partitions a whole data space into a small number of disjoint regions with the aid of a dynamical system. Finally, it labels the decomposed regions utilizing the labeled data and the cluster structure described by the constructed support function. Simulation results show the effectiveness of the proposed method to label out-of-sample unlabeled test data as well as in-sample unlabeled data.

A new learning algorithm for semi-supervised learning is proposed. The proposed method utilizes a support vector machine to describe domains and a dynamical system to decompose the data space into several labelled disjoint regions. It can classify unlabelled data and predict new unknown data. Effectiveness of the method is veri£ed through simulation results.

A novel active learning method for security assessment of nonlinear systems is presented. The proposed method first extracts a dataset near the stability region boundary by using the direct method, and then learns a SVM model from the data. The constructed SVM classifier is shown to dramatically reduce the conservativness of the estimated stability region and also to make a fast security assessment.

A topological and dynamical characterization of the cluster structures described by the support vector clustering is developed. It is shown that each cluster can be decomposed into its constituent basin level cells and can be naturally extended to an enlarged clustered domain, which serves as a basis for inductive clustering. A simplified weighted graph preserving the topological structure of the clusters is also constructed and is employed to develop a robust and inductive clustering algorithm. Simulation results are given to illustrate the robustness and effectiveness of the proposed method.

The support vector clustering (SVC) algorithm is a recently emerged unsupervised learning method inspired by support vector machines. One key step involved in the SVC algorithm is the cluster assignment of each data point. A new cluster labeling method for SVC is developed based on some invariant topological properties of a trained kernel radius function. Benchmark results show that the proposed method outperforms previously reported labeling techniques.

The concept of a stability region (region of attraction) of nonlinear dynamical systems is widely used in many fields such as engineering and the sciences. In this paper, we study the notion of stability regions for a class of non-hyperbolic dynamical systems. A complete characterization of the stability boundary is presented for a fairly large class of non-hyperbolic dynamical systems. Several necessary and sufficient conditions for an equilibrium manifold (the generalized concept of an equilibrium point) to lie on the stability boundary are derived. It is shown that the stability boundary of this class of systems consists of the union of the stable manifolds of the equilibrium manifolds on the stability boundary. In addition, an effective scheme to estimate stability regions by using an energy function is developed. It is shown that the scheme can optimally estimate stability regions for a class of non-hyperbolic dynamical systems. Two examples are given to illustrate the theoretical prediction.

This paper introduces the concept of the convergent region of the Newton homotopy method. It is shown that convergent regions of the Newton homotopy method are equal to the stability regions for the Newton flow x/spl dot/=-adj(DF(x))F(x). A quite complete algebraic characterization of a convergent region and its boundary for a large class of nonlinear systems is derived and this characterization, which is explicit and computationally feasible, leads to the development of a numerical method to determine the convergent region and to the construction of simple criteria to avoid revisitations of the same solutions from different initial guesses. Two examples are given to illustrate the theoretical prediction.

A comprehensive analysis of homotopy methods on the computation of all DC operating points of nonlinear circuits and systems is conducted. Several sufficient conditions for the connectivity of all the solutions along a single homotopy path are derived. These conditions offer criteria to determine a starting point from which one can find all the solutions along one homotopy path. For the class of nonlinear circuits and systems in which all the solutions lie on several homotopy paths, a new systematic method to explicitly construct a starting point for each homotopy path is developed, From a practical viewpoint, the constructive method developed does not require the difficult task of finding a good initial guess and is applicable to general nonlinear circuits and systems. From a methodological viewpoint, the constructive method developed is applicable to general homotopy methods with different homotopy functions.