In this paper, we propose a deep learning based breadth-first sphere decoding (SD) scheme to reduce the detection complexity for multiple-input multiple-output (MIMO) communication systems. Specifically, we first design the DenseNet-based deep neural network (DN-DNN) to provide the pruning threshold for SD at each layer. Then, we develop modified number-based SD (MNSD) to reduce the complexity of SD by constraining the number of visited nodes at each layer with the output of DN-DNN. We use a distance-based SD (DSD) to further reduce the complexity of MNSD by constraining the accumulated distance at each layer with the output of DN-DNN. Compared with the traditional M-best SD withM= 16, the proposed MNSD achieves similar performance but reduces about 25% complexity for QPSK modulation; the proposed DSD has better performance with up to 75% complexity reduction at the high SNR region for 16QAM.

In this paper, we propose a deep learning based sphere decoding (SD) scheme to reduce the detection complexity for the multiple-input multiple-output (MIMO) communication systems. Specifically, we first design the sparsely connected deep neural network (SC-DNN) to find a moderate radius for the SD algorithm. Then, we develop the SC-SD algorithm to reduce the computational complexity by deciding the detection order from the output of the SC-DNN, the zero-forcing (ZF) detector, and the transmit power. We further reduce the complexity of the SC-SD by defining partial layers without searching. For multi-stream MIMO, where a large number of parameters in neural networks should be trained, we propose a partitioned training procedure to achieve a reasonable computational complexity. Simulation results demonstrate that the SC-SD almost achieves the performance of the maximum likelihood (ML) in MIMO system but is much faster than the classic SD algorithm.