Machine learning systems require massive amounts of labeled training data in order to achieve high accuracy rates. Active learning uses feedback to label the most informative data points and significantly reduce the training set size. Many heuristics for selecting data points have been developed in recent years which are usually tailored to a specific task and a general unified framework is lacking. In this work, the individual setting is considered and an active learning criterion is proposed. Motivated by universal source coding, the proposed criterion attempts to find data points which minimize the Predictive Normalized Maximum Likelihood (pNML) regret on an un-labelled test set. It is shown that for binary classification and linear regression, the resulting criterion coincides with well known active learning criteria and thus represents a unified information theoretic active learning approach for general hypothesis classes. Finally, it is shown using real data that the proposed criterion performs better than other active learning criteria in terms of sample complexity.

In this paper we consider the problem of on-line learning in the stochastic setting under a certain form of hypotheses class. We prove an equivalence between the minimax redundancy and capacity of the channel between the class parameters and the labels conditioned on the data features (side information). Our proof extends Gallager's Redundancy Capacity theorem for universal prediction to on-line learning with the considered form of hypotheses class. Moreover, this result confirms the optimality of previous ad-hoc universal learners, or universal predictors with side information, but more importantly, extends these previous results to more general hypotheses classes.

Phase noise poses a serious challenge for high-speed digital communications systems mainly when going to higher and higher carrier frequencies, such as in satellite communications. Traditionally, phase noise estimation was performed separately from the decoding task and it was shown, recently, that there is much to be gained from joint estimation and decoding, particularly when using LDPC (low-density parity check)/turbo codes. However, jointly estimating phase noise and decoding is a very complex and computationally demanding task. In this paper, we propose several algorithms based on the sum and product algorithm (SPA) for low complexity joint decoding and estimation of coded information in strong phase noise channels. These algorithms are based on a novel approximation of SPA messages as Tikhonov mixtures of a given order. Since mixture-based Bayesian inference such as SPA, creates an exponential increase in mixture order for consecutive messages, a mixture reduction scheme is a must. Therefore, in this paper, we propose a low complexity mixture reduction algorithm, which provably satisfies an upper bound on the Kullback Leibler (KL) divergence between the mixture and the reduced mixture. We then reduce the complexity even further, including limiting the model order and reducing the clustering effort to simple component selection. As an extreme case, it is even possible to reduce the number of modes to one. We show the relation between the simplified algorithm to the phase locked loop (PLL). Finally, we show simulation results and complexity analysis for the proposed algorithms, which show superior performance over other state of the art low complexity algorithms.

In this contribution, we propose a signal constellation for the the phase noise channel which does not require pilots thus increasing the effective information rate in a communication system. This constellation does not present rotational symmetry thus enabling decoding algorithms such as SPA, to converge without the use of pilots. We will provide Bit Error Rate (BER) simulations which show the superiority of this constellation over standard MPSK with pilots. Moreover, we will provide a method to analyze any arbitrary signal constellation and provide a figure of merit for its performance when iterative decoding algorithms are used.