Utilizing the multi-dimensional (MD) space for constellation shaping has been proven to be an effective approach for achieving shaping gains. Despite there exists a variety of MD modulation formats tailored for specific optical transmission scenarios, there remains a notable absence of a dependable comparison method for efficiently and promptly re-evaluating their performance in arbitrary transmission systems. In this paper, we introduce an analytical nonlinear interference (NLI) power model-based shaping gain estimation method to enable a fast performance evaluation of various MD modulation formats in coherent dual-polarization (DP) optical transmission system. In order to extend the applicability of this method to a broader set of modulation formats, we extend the established NLI model to take the 4D joint distribution into account and thus able to analyze the complex interactions of non-iid signaling in DP systems. With the help of the NLI model, we conduct a comprehensive analysis of the state-of-the-art modulation formats and investigate their actual shaping gains in two types of optical fiber communication scenarios (multi-span and single-span). The numerical simulation shows that for arbitrary modulation formats, the NLI power and relative shaping gains in terms of signal-to-noise ratio can be more accurately estimated by capturing the statistics of MD symbols. Furthermore, the proposed method further validates the effectiveness of the reported NLI-tolerant modulation format in the literature, which reveals that the linear shaping gains and modulation-dependent NLI should be jointly considered for nonlinearity mitigation.

Multidimensional Voronoi constellations (VCs) have been shown to be more power-efficient than quadrature amplitude modulation (QAM) formats given the same uncoded bit error rate, and also have higher achievable information rates. However, a coded modulation scheme that sustains these gains after forward error correction (FEC) coding is still lacking. This paper designs coded modulation schemes with soft-decision FEC codes for VCs, including bit-interleaved coded modulation (BICM) and multilevel coded modulation (MLCM), together with three bit-to-integer mapping algorithms and log-likelihood ratio calculation algorithms. Simulation results show that VCs can achieve up to 1.84 dB signal-to-noise ratio (SNR) gains over QAM with BICM, and up to 0.99 dB SNR gains over QAM with MLCM for the additive white Gaussian noise channel at the bit error rate of$1.81\times 10^{-3}$, with a low decoding complexity.

Voronoi constellations (VCs) are finite sets of vectors of a coding lattice enclosed by the translated Voronoi region of a shaping lattice, which is a sublattice of the coding lattice. In conventional VCs, the shaping lattice is a scaled-up version of the coding lattice. In this paper, we design low-complexity VCs with a cubic coding lattice of up to 32 dimensions, in which pseudo-Gray labeling is applied to minimize the bit error rate. The designed VCs have considerable shaping gains of up to 1.03 dB and finer choices of spectral efficiencies in practice compared with conventional VCs. A mutual information estimation method and a log-likelihood approximation method based on importance sampling for very large constellations are proposed and applied to the designed VCs. With error-control coding, the proposed VCs can have higher information rates than the conventional scaled VCs because of their inherently good pseudo-Gray labeling feature, with a lower decoding complexity.

In a classical 1983 paper, Conway and Sloane presented fast encoding and decoding algorithms for a special case of Voronoi constellations (VCs), for which the shaping lattice is a scaled copy of the coding lattice. Feng generalized their encoding and decoding methods to arbitrary VCs. Less general algorithms were also proposed by Kurkoski and Ferdinand, respectively, for VCs with some constraints on their coding and shaping lattices. In this work, we design VCs with a cubic coding lattice based on Kurkoski's encoding and decoding algorithms. The designed VCs achieve up to 1.03 dB shaping gains with a lower complexity than Conway and Sloane's scaled VCs. To minimize the bit error rate (BER), pseudo-Gray labeling of constellation points is applied. In uncoded systems, the designed VCs reduce the required SNR by up to 1.1 dB at the same BER, compared with the same VCs using Feng's and Ferdinand's algorithms. In coded systems, the designed VCs are able to achieve lower BER than the scaled VCs at the same SNR. In addition, a Gray penalty estimation method for such VCs of very large size is introduced.