The well-known Bahl-Cocke-Jelinek-Raviv (BCJR) algorithm is used as a minimum mean-squared error (MMSE) decoder for combined trellis-coded quantization and modulation. When the BCJR algorithm is used as decoder, the optimal trellis-coded encoder is derived. Both trellis-coded quantizer and the soft-decoder can be optimized iteratively for minimum overall distortion from transmitter input to receiver output. The optimal encoder/decoder is simulated for Rayleigh fading channel. Significant performance improvement is achieved for both memoryless Gaussian and uniform source, especially for very noisy channel.

The maximum a posteriori (MAP) algorithm is used as a minimum mean-squared-error decoder for combined trellis-coded quantization and modulation. The optimal trellis-coded quantizer is derived when the soft-decoding MAP algorithm is used as a decoder. The trellis-coded quantizer and the soft decoder are optimized iteratively for minimum overall distortion from transmitter input to receiver output. Significant performance improvement is achieved for both memoryless Gaussian and uniform source, especially for very noisy channels.