Abstract In this paper, a three‐user non‐orthogonal multiple access (NOMA) system based on device‐to‐device (D2D) communication and full‐duplex (FD) transmission is proposed, in which the D2D user acts as a decode‐and‐forward (DF) relay to forward the signal from the base station (BS) to the far user, forming a system termed the FD D2D transmitter aided NOMA (FD‐D2D‐NOMA). In practical NOMA systems, due to the limitation of decoding ability, the near user cannot always perform successive interference cancellation (SIC) successfully. By considering perfect SIC (pSIC) and imperfect SIC (ipSIC), the close‐form expressions for the ergodic sum capacity and outage probability of the proposed FD‐D2D‐NOMA scheme are derived, which match well with the simulation ones. The ergotic sum capacity and outage probability under different levels of residual self‐interference at the FD user and residual interference at the far/near user due to ipSIC are investigated, and the results show that, such interferences have significant impact on the performance of the proposed scheme. Moreover, the analytical and simulation results illustrate the performance gain of the proposed scheme compared with other ones under both pSIC and ipSIC scenarios. The impacts on performance exerted by channel conditions and design parameters are also discussed in this paper.

Abstract In non‐orthogonal multiple access (NOMA) systems, it may not be advisable to send the superimposed signal of all users to individual users with retransmission requirements during the retransmission phase. This paper investigates a novel cognitive radio network based on hybrid NOMA/orthogonal multiple access (CR‐H‐NOMA/OMA) in underlay spectrum sharing, whereby a hybrid NOMA/OMA cooperative mechanism is proposed for the secondary network by considering that the access modes of secondary users (SUs) and relays can be switched between NOMA and OMA. In order to compensate the loss of throughput caused by successive interference cancellation (SIC) during the cooperative phase, one or two relays with the best channel quality between themselves and the destination users are selected according to the states of the initial signal decoded by SUs, using truncated automatic repeat request (T‐ARQ) to retransmit only the desired signal of SU instead of the entire superimposed signal. Exact closed‐form expressions for outage probabilities of SUs and system throughput of the secondary network are derived under the interference constraint of the primary network to evaluate the performance of the proposed cooperative scheme. Numerical and simulation results demonstrate the superiority of the proposed hybrid NOMA/OMA cooperative scheme when compared with conventional pure NOMA and pure OMA based on time division multiple access (TDMA) in terms of secondary throughput. Also, the outage and throughput performance depend critically on the choices of key paraments such as power allocation factor and target rate.