Clustering is a data mining technique intensively used for data analytics, with applications to marketing, security, text/document analysis, or sciences like biology, astronomy, and many more. Dirichlet Process Mixture (DPM) is a model used for multivariate clustering with the advantage of discovering the number of clusters automatically and offering favorable characteristics. However, in the case of high dimensional data, it becomes an important challenge with numerical and theoretical pitfalls. The advantages of DPM come at the price of prohibitive running times, which impair its adoption and makes centralized DPM approaches inefficient, especially with high dimensional data. We propose HD4C (High Dimensional Data Distributed Dirichlet Clustering), a parallel clustering solution that addresses the curse of dimensionality by two means. First it gracefully scales to massive datasets by distributed computing, while remaining DPM-compliant. Second, it performs clustering of high dimensional data such as time series (as a function of time), hyperspectral data (as a function of wavelength) etc. Our experiments, on both synthetic and real world data, illustrate the high performance of our approach.

The well-known cumulative sum (CUSUM) sequential rule for abrupt model change detection in stochastic dynamic systems relies on the knowledge of the probability density functions of the system output variables conditional on their past values and on the system functioning mode at each time step. This paper shows how to build an asymptotically optimal detection rule under the common average run length (ARL) constraint when these densities are not available but can be consistently estimated. This is the case for nonlinear state–space systems observed through output variables: for such systems, a new class of particle filters based on convolution kernels allows to get consistent estimates of the conditional densities, leading to an optimal CUSUM-like filter detection rule (FDR).