Abstract Transport Layer Cryptography Protocol (TLCP) is a secure communication protocol developed in China, featuring a dual-certificate architecture and incorporating ShangMi cryptographic algorithms. It has been widely deployed in security-critical domains such as finance, government, and energy. Despite its practical significance, TLCP did not undergo comprehensive formal analysis during its standardization process, leaving potential design-level vulnerabilities insufficiently explored. Moreover, the advent of quantum computing poses fundamental challenges to the classical cryptographic primitives employed by TLCP, motivating the need for both systematic security evaluation and post-quantum enhancements. To address these gaps, we first construct the comprehensive formal model of TLCP, covering certificate-based and identity-based cipher suites as well as its distinctive dual-certificate mechanism, under a realistic threat model and security assumptions that capture both classical and quantum adversaries. Based on this model, we conduct an automated security analysis using ProVerif, identifying nine potential attack vectors and deriving five concrete mitigation recommendations. Finally, motivated by the analysis results and the limitations of incremental fixes against quantum threats, we propose KEMTLCP, a post-quantum secure variant of TLCP that leverages key encapsulation mechanisms (KEMs) for both key exchange and authentication while preserving TLCP’s architectural principles through a novel explicit authentication mechanism. We further provide a security proof for the core authentication mechanism, show that KEMTLCP effectively mitigates the majority of identified vulnerabilities through formal analysis, and evaluate its practical performance.

We propose a new physical-layer public key encryption scheme and establish a trapdoor one-way function through Massive MIMO techniques and precoding designs. Under standard arguments, we show that the eavesdropper's decoding complexity grows exponentially with the number of antennas, while the legitimate receiver's decoding complexity grows only quadratically. The proposed scheme builds a bridge between information-theoretic security and cryptographic security. Compared to the traditional physical-layer security, the proposed scheme is secure when the number of the eavesdropper's antennas is infinite or much larger than the number of transmitter/receiver antennas, provided that the eavesdropper's distance from the legitimate receiver is less than one-half of the wavelength, or that the channel estimation process between the sender-receiver pair is broken by the eavesdropper. Because the scheme is based on lattice, not on channel reciprocity, it can be applied to both time-division duplex and frequency-division duplex channels, and utilizes the simple physical layer characteristics of Massive MIMO to resist the currently known quantum attacks. The proposed scheme is adapted to the future requirements of 6G for the security of communication, and provides a new idea for the post-quantum cryptosystem. The simulation results show that the proposed scheme has a decoding bit error rate (BER) close to 0.5 at the eavesdropper and almost 0 at the legitimate receiver.