In this paper we specify an optimal receiver filter for 2phase PSK transmission over a channel consisting of the tandem connection of a linear filter and a bandpass nonlinearity, with thermal noise added at the channel output. This filter minimizes the bit estimation error subject to the constraint that it be a linear filter. Our main aim is to consider satellite communication channels and as such the system nonlinearity is taken to have both AM/AM and AM/PM conversions. In these systems we assume the up-link signal-to-noise ratio is large and thus our work relates to large transmit terminal applications. In an example the probability of error performance of the optimal received filter is found to be 2 dB better in terms of the effective output SNR as compared with that of a standard choice for the receiver filter.

The problem of designing and evaluating the performance of a maximum likelihood sequence receiver for binary PSK transmission over bandlimited nonlinear channels is considered in this paper. The effects of intersymbol interference followed by AM/AM and AM/PM conversions are taken into account while optimizing the performance in the presence of white Gaussian noise. A new representation for the output of a bandpass nonlinearity is given when the input consists of a carrier signal modulated by a sum of binary overlapping pulses. The structure of a maximum likelihood sequence receiver for a bandlimited nonlinear channel is derived using this representation. The receiver uses a modified Viterbi algorithm to determine the most likely sequence of data symbols transmitted. An upperbound on the probability of symbol error for this receiver is obtained. Numerical results illustrating the applicability of the present work to optimizing the performance of a digital satellite communications link are also presented.