This paper derives consistency results for estimation in the finite direct sum of reproducing kernel Hilbert spaces (RKHS) for dependent data. The link between penalized and constrained estimation is established. We consider the relation between topological equivalent norms for direct sums of RKHS. These norms have different implications for estimation. Estimation in a ball of the RKHS defined by these norms essentially results in estimation with a ridge and Lasso penalty, respectively. A greedy algorithm for the solution of the estimation problem under these two norms is discussed for general loss functions.

We consider stationary autoregressive processes with coefficients restricted to an ellipsoid. These are included in the family of autoregressive processes with absolutely summable coefficients. We provide consistency results under different norms for the estimation of such processes using constrained and penalized estimators. As an application, we show a weak form of universal consistency. Simulations show that directly including the constraint in the estimation can lead to results that are more robust.

Recursive algorithms for the estimation of mixtures of densities have attracted a lot of attention in the last 10 years. Here an algorithm for recursive estimation is studied. It complements existing approaches in the literature, as it is based on conditions that are usually very weak. For example, the parameter space over which the mixture is taken does not need to be necessarily bounded. The essence of the procedure is to combine density estimation via empirical characteristic function together with an iterative Hilbert space approximation algorithm. The conditions for consistency of the estimator are verified for three important statistical problems. A simulation study is also included.