We analyze the performance of the error probability and the mutual information for quadrature amplitude modulation (QAM) and phase-shift keying systems based on quantum detection theory. It is shown that the quantum receiver called square root measurement gives about 5.7 dB improvement in power in comparison with the classical one. Furthermore, we show that the quantum QAM system can achieve the same reliability as the conventional intensity modulation-direct detection (IM-DD) system with almost equal power, compressing the required bandwidth, while the reliability is degraded in general.