This paper investigates the generalized linear system (GLS), widely employed to evaluate the impact of nonlinear preprocessing on wireless transceivers. Two state-of-the-art signal recovery algorithms, namely generalized approximate message passing (GAMP) and generalized orthogonal/vector AMP (GOAMP/GVAMP), are comparatively studied. They have demonstrated Bayesian optimality for independently and identically distributed (IID) Gaussian matrices and unitary matrices, respectively. However, Bayesian optimality does not inherently guarantee error-free signal recovery. For coded GLS, the information-theoretic (i.e., achievable rate) limit of GAMP remains unknown, and there are still no analytical comparisons between GAMP and GOAMP/GVAMP in terms of the mean-square error and information-theoretic limit. To address these issues, we present the achievable rate analysis and optimal coding principle for GAMP with IID Gaussian matrices, as well as provide comprehensive comparisons with GOAMP/GVAMP with unitary matrices. Specifically, based on the celebrated I-MMSE lemma and the preconditions for state evolution (SE) to hold, the simplified variational SEs of GAMP and GOAMP/GVAMP are derived, leveraging the IID and unitary matrix properties to analyze the achievable rate and optimal coding principle, respectively. On this basis, it is proven that GOAMP/GVAMP outperforms GAMP in terms of asymptotic MSE and maximum achievable rate while requiring less complexity. Furthermore, two common nonlinear functions, clipping and quantization, are used as examples to demonstrate the theoretical comparisons and practical low-density parity-check (LDPC) code design for GAMP and GOAMP/GVAMP. Numerical results show that GAMP and GOAMP/GVAMP with optimized LDPC codes can approach the theoretical limits within 0:3 dB and overcome the decoding deterioration and even divergence of the existing state-of-the-art methods, particularly under low-resolution quantization.