This paper describes a new interpretation of the maximum-likelihood sequence detector metric for a linearly modulated signal and an unknown time-varying, dispersive, Rayleigh fading channel. The metric, which involves the channel and noise autocovariances, is transformed into the intuitive Mahalanobis distance (squared Euclidean distance in white noise) between the observations and a reference estimated from them. A novel estimator, called the autocovariance preserving estimator, is obtained that trades off increased estimation bias for reduced sequence-error probability.

By using the innovations process, this paper provides a unification and extension of the existing maximum a posteriori (MAP) detectors (MAPDs). The practically important topics of linear modulations, time-varying frequency-selective channels, differential phase detection, and fractional sampling are accounted for. The MAPDs are derived under different conditions of optimality and a priori knowledge as follows: when the MAP criterion is applied to the constellation mapper's input bits or output symbols, when all observations or only a fixed number of future observations (i.e. fixed-lag MAPDs) from a transmission are available, when the time-varying channel impulse response is perfectly known, and when only the Gaussian-distributed channel's mean and autocovariance and the noise variance are known. As these quantities are actually unknown, their estimation in the context of MAP detection is also discussed. The MAPDs are characterized through simulation and a novel, unified analysis. Although MAPDs are less suited to hardware implementation than the traditional maximum-likelihood sequence detectors, the MAPDs can accept nonuniform a priori bit or symbol probabilities and provide soft outputs. In this way, the MAPDs are well suited to iterative decoding, and so they will become increasingly integral to high-performance receiver designs.

Several receiver structures for linearly modulated signals are proposed for time-varying frequency-selective channels. Their channel estimators explicitly model the time variation of the channel taps via polynomials. These structures are constructed from the following building blocks: (i) sliding or fixed block channel estimators; (ii) maximum likelihood sequence detectors (MLSDs) or decision feedback equalizers (DFEs); and (iii) single or multiple passes. A sliding window channel estimator uses a window of received samples to estimate the channel taps within or at the end of the window. Every symbol period, the window of samples is slid along another symbol period, and a new estimate is calculated. A fixed block channel estimator uses all received samples to estimate the channel taps throughout the packet, all at once. A single pass receiver estimates the channel and detects data once only. A multipass receiver performs channel estimation and data detection repetitively. The effect of the training symbol positions on the performance of the block multipass approach is studied. The bit error rate (BER) performance of the MLSD structures is characterized through simulation and analysis. The proposed receivers offer a range of performance/complexity tradeoffs, but all are well suited to time-varying channels. In fast fading channels, as the signal-to-noise ratio (SNR) increases, they begin to significantly outperform the per-survivor processing-based MLSD receivers which employ the least mean-squares (LMS) algorithm for channel estimation.

A receiver structure for unknown time-varying, frequency-flat, Rayleigh fading channels is proposed. Unlike the maximum-likelihood receiver structure, this receiver does not need any knowledge of the channel autocovariance or the received signal-to-noise ratio, and yet analytic and simulation results show, that its bit-error rate performance is close to the optimal structure. The receiver is applicable to signals with constant or approximately constant envelopes or to linearly modulated signals using Nyquist pulses.

A rectangular-windowed least-squares estimator using a polynomial model of the time-varying channel taps is proposed for estimating the impulse response of a frequency-selective fading channel. This method provides a significant improvement in mean square identification error (MSIE) over the conventional least mean squares (LMS) and the exponentially weighted recursive least squares (EW-RLS) algorithms without a polynomial model. A detailed study of the effects of channel parameters, such as the fading rate and the signal-to-noise ratio, on the proposed method is carried out. The performance of the method depends on the window size of the least squares estimator and the polynomial order being used. Algorithms to obtain the approximately optimal window size and polynomial order are proposed and are shown to perform well.

Yu and Pasupathy (see ibid., vol.43, p. 1534-44, 1995) derived a maximum-likelihood sequence estimation (MLSE) receiver structure for unknown time varying frequency-selective Rayleigh-fading channels and uncorrelated diversity sources. This receiver design is extended to the case of correlated diversity sources. Correlated diversity sources typically arise with space diversity, where constraints on antenna volume require that diversity antennae be placed too closely together. Analytic and simulated bit-error rate (BER) curves are presented for receivers which exploit and ignore the correlation. In the former case, we find a small BER improvement that reduces with decreasing correlation. However, for a fixed receiver complexity, superior performance is achieved when the correlation is ignored.

This paper develops a receiver structure to perform jointly maximum-likelihood (ML) synchronization, equalization, and detection of a linearly modulated signal transmitted over a time-varying frequency-selective Rician-faded channel, corrupted by additive white Gaussian noise (AWGN). The receiver is particularly suited to a fast-fading channel, where other receivers that rely on estimating the channel cannot track it quickly enough. The signal mean and autocovariance are needed, and a scheme is proposed for estimating these quantities adaptively. The receiver processes the specular and diffuse components (corresponding to the signal mean and autocovariance) separately. Processing the known specular component is the classical detection problem. The unknown diffuse component is processed by predictors. We show that the predictors can achieve synchronization in a novel manner, if synchronization is required. A union bound on the receiver's bit-error rate (BER) is derived, and it tightly bounds simulated BERs in fast-fading at high signal-to-noise ratios (SNRs).

This paper develops a maximum-likelihood sequence estimation (MLSE) diversity receiver for the time- and frequency-selective channel corrupted by additive Gaussian noise when linear constellations (M-ASK, M-PSK, M-QAM) are employed. The paper extends Ungerboeck's derivation of the extended MLSE receiver for the purely frequency-selective channel to the more general channel. Although the new receiver structure and metric assume ideal channel-state information (CSI), the receiver can be used wherever high-quality CSI is available, such as a comb of pilot tones or time-isolated symbols. The major contributions of this paper are as follows: (1) the derivation of a finite-complexity diversity receiver that is maximum likelihood (ML) for all linear channel models and sources of diversity, as long as ideal CSI is available; (2) a benchmark, in that the new receiver's performance is a lower bound on the performance of practical systems, which either lack ideal CSI or are not ML; (3) insight into matched filtering and ML diversity receiver processing for the time- and frequency-selective channels; and (4) bounds on the new receiver's bit-error rate (BER) for ideal CSI and pilot tone CSI, in a fast Rayleigh-fading channel with multiple independently faded paths. The new receiver can seamlessly tolerate square-root Nyquist pulses without a fading-induced ISI error floor.