There has been increased interest in devising learning techniques that combine unlabeled data with labeled data ? i.e. semi-supervised learning. However, to the best of our knowledge, no study has been performed across various techniques and different types and amounts of labeled and unlabeled data. Moreover, most of the published work on semi-supervised learning techniques assumes that the labeled and unlabeled data come from the same distribution. It is possible for the labeling process to be associated with a selection bias such that the distributions of data points in the labeled and unlabeled sets are different. Not correcting for such bias can result in biased function approximation with potentially poor performance. In this paper, we present an empirical study of various semi-supervised learning techniques on a variety of datasets. We attempt to answer various questions such as the effect of independence or relevance amongst features, the effect of the size of the labeled and unlabeled sets and the effect of noise. We also investigate the impact of sample-selection bias on the semi-supervised learning techniques under study and implement a bivariate probit technique particularly designed to correct for such bias.

We introduce a mixture-of-experts technique that is a generalization of mixture modeling techniques previously suggested for semi-supervised learning. We apply the bias-variance decomposition to semi-supervised classification and use the decomposition to study the effects from adding unlabeled data when learning a mixture model. Our empirical results indicate that the biggest gain from adding unlabeled data comes from the reduction of the model variance, whereas the behavior of the bias error term heavily depends on the correctness of the underlying model assumptions.