To enhance the performance of pre-trained foundation models in various natural language processing tasks, the scale of these models has been steadily increasing in recent years, driven by advancements high-performance computing hardware. However, the presence of diverse computing devices in data centers and the varying computational demands of different tasks make it challenging for a single type of high-performance G PU to suffice for the large-scale training of foundation models' and lower-performance GPUs often remain underutilized. Therefore, optimizing resourceallocationand improving training efficiency have become a critical challenge. To address these issues, this study proposes a hybrid parallel training framework for heterogeneous computing devices, named HyperPipe. This framework automatically selects the optimal hybrid parallel strategy based on the performance characteristics of the devices, thereby achieving efficient resource utilization and acceleration training to meet the computational demands of large-scale FMs. Experimental results demonstrate that in a heterogeneous computing environment, training large models like GPT-3-1.3B with HyperPipe achieves significant acceleration, delivering speed-ups ranging from 1.6 x to 2.8 x compared to previous methods.

With the rapid development of large-scale Chinese pre-trained language models and natural language processing technologies, the demand for large, high-quality Chinese corpora is significantly increasing. However, current Chinese corpora often fail to meet the demands of high-quality training due to their limited size and substandard quality. This paper introduces a method to extract high-quality Chinese corpora based on the Common Crawl dataset, enhancing existing quality filtering and deduplication methods. In terms of deduplication, this paper enhances the Simhash algorithm to tackle the issue of feature word co-occurrence effectively and introduces a deduplication method based on sentence clusters. Additionally, this paper proposes an offensive speech detection method, ODet, based on prefix tuning and prompt learning. ODet has demonstrated superior performance compared to existing offensive speech detection methods and has been successfully applied in the corpus processing workflow. We successfully extracted a 500GB high-quality Chinese corpus, CHCorpus, employing this approach. Furthermore, we applied a perplexity calculation method to segment CHCorpus into head, middle, and tail sections. The experimental outcomes indicate that CHCorpus excels in perplexity assessments, several CLUE benchmark tasks, and text summarization, surpassing Wikipedia, CLUECorpus2020, and CCNet corpora. This research provides a robust data foundation for training large-scale Chinese pre-trained models, significantly improving their robustness and security.

Cost-effective hybrid precoding has been attracting extensive attention for millimeter wave full-dimension multiple input multiple output (FD-MIMO) green communication systems. To enhance beamforming flexibility and energy efficiency (EE) while obtaining satisfactory spectral efficiency (SE), this paper introduces an adaptive fully-connected subarrays structure with two-layer shared phase shifters (TPSAS) and proposes a TPSAS-based two-stage double dimensional hybrid precoding (TSDDHP) algorithm. Firstly, the TPSAS structure is given to reduce hardware complexity while attractively making full use of the vertical and horizontal dimensions information. Then, to achieve TPSAS-based energy-efficient and flexible hybrid precoding, the TSDDHP algorithm is developed to obtain digital and analog precoders for point-to-point (PTP) and multi-user (MU) scenarios. For PTP scenario, the vertical analog precoder, the horizontal analog precoder, and the digital precoder are acquired by the equivalent quadratic function, properties of dual norm and unitarily invariant norm, respectively. For MU scenario, a linear zero-forcing approach and an equivalent quadratic function are used to gain the digital, the vertical and horizontal analog precoders, respectively. Theoretical analyses and simulation results verify that the proposed scheme is superior to others in terms of beamforming flexibility and EE while acquiring satisfactory SE with lower computation complexity.

The Internet of Things data (IoT) with noise and substantial outliers are unavoidable, due to imperfect communication environment. Tensor decomposition (TD)-based robust principal component analysis (TD-RPCA) methods have achieved effective IoT data recovery performance by capturing the global information when the corresponding TD rank is accurately estimated. That is, their recovery performance may rely on the predefined TD rank. Therefore, this article proposes a new TRPCA data recovery method from the coexistence of noise and outliers even without accurate rank estimation. In the proposed method, the IoT data recovery problem is transformed into a defined multimode multiple relationships nuclear norm (MRNN) minimization TRPCA optimization problem. Specifically, the MRNN is constructed by CP decomposition (CPD) factor matrix (FM) with Hankel factor, which can simultaneously extract both adjacent and distant data relationships in the CPD-FM along different modes, so as to capture additional global information for improving recovery performance. Besides, theoretical analysis also proves that the defined MRNN can not only weaken the effect of predefined rank, but also balance the low-rank feature along different mode for CPD. Moreover, an entropy regularization term is constructed by the probabilities of occurrence between outliers and noise in the proposed method, which can enhance the recovery performance by effectively demonstrating the different effects of them and reducing their classification bias. Theoretical analysis and simulation results show that the proposed method has lower computational complexity and better restoration performance compared with other methods even the outlier ratio is large in various IoT applications.

As one of the promising technologies for the forthcoming 6G millimeter-wave massive multiple-input multiple-output (MIMO) orthogonal frequency division multiplexing (OFDM) systems, hybrid precoding/combining can achieve trade-off between the achievable rate, the power consumption, and cost. In this paper, a two-stage joint hybrid precoding/combining algorithm based on masterful row and column selection (MCUR) decomposition is proposed for the MIMO-OFDM system with subsystems, named as MCUR-TS. During the first stage, the analog precoder/combiner are found by the MCUR and viewpoint of subsystems, which includes the analog precoding/combining subsystems. Then, using the MCUR and notion of equivalent channel, the digital precoder/combiner are acquired. Different from the existing singular value decomposition (SVD)-based algorithms, the proposed MCUR-based algorithm preserves the sparsity of original data while reducing the computational complexity. Meanwhile, the MCUR with the truncated rotated orthogonal triangular factorization and the gravity centers of column subspaces enhances the performance of hybrid precoding/combining. Furthermore, the phase shifter quantization and imperfect channel state information are applied to decrease energy consumption and improve practicality, respectively. Extensive simulations and theoretical analyses illustrate that the proposed algorithm outperforms other methods in terms of spectral efficiency and achievable sum rate with lower complexity while providing with better robustness to channel errors.

Hybrid precoding is considered as a promising candidate technology for millimeter wave (mmWave) massive multiple-input multiple-output (MIMO) systems. Conventional hybrid precoder with fixed fully-connected or sub-connected (SC) architectures often requires lots of phase shifters (PSs) with considerable power consumption. To address the challenge, this paper firstly presents an adaptive SC (ASC) hybrid precoding architecture, which dynamically divides the antennas elements into several subsets according to the channel environments, and each subset is connected to all RF chains with a small number of PSs. Owing to the flexible division and much fewer PSs, the proposed ASC architecture is expected to achieve a high spectral efficiency with lower power consumption in comparison to the conventional SC architecture. Afterwards, an alternative optimization hybrid precoding algorithm is proposed based on the ASC architecture. The proposed algorithm optimizes the switch precoding matrix via enumerations, and then an elegant diagonal loading factor is derived to reformulate the objective function of the PS precoding matrix as a positive semidefinite quadratic form, so that the PS precoding matrix can be derived easily. Simulation results show that the proposed hybrid precoding architecture and algorithm can remarkably improve the energy efficiency when maintaining a high spectral efficiency compared with the conventional counterpart.

In cognitive radio (CR), the problem of spectrum hole detection has been extensively studied in single dimension, such as frequency domain, spatial domain, and so on. Recently, a class of two dimension spectrum hole detection methods, named as joint angle-frequency estimation (JAFE), has attracted much attention. Nevertheless, most of the existing approaches are only suitable for the scenario with matching frequency-angle pairs, rather than the non-matching scenario between the two parameters, like space division multiple access (SDMA) or frequency division multiplexing access (FDMA) communication mode which allows the same frequency band (or angle) to be reused in different angles (or frequencies). For the above two cases, including matching and non-matching scenarios, this paper develops an effective frequency-angle spectrum hole detection algorithm with Taylor expansion based focusing transformation (TFT-FASHD), on the basis of signal sparse representation by extending the array manifold from angle domain to frequency-angle domain. In the proposed method, for Fourier transform representation of the sparse model, a focusing transformation based on Taylor expansion is first performed to focus the signal subspaces at different frequencies to a single frequency, so as to carry out dimension reduction of dictionary in angular domain. TFT is derived by decomposing the array manifold with Taylor expansion, and further the optimum focusing frequency of focusing transform is discussed theoretically. Second, atoms with high representative performance are chosen by the presented TFT and compressed sensing (CS). Third, according to the low dimension dictionary, the TFT-FASHD is implemented by CS under multiple measurement vector (MMV) circumstances. The accuracy of the algorithm in non-matching scenario is verified by simulation results. For the matching scenario, compared with the related JAFE methods, the proposed algorithm has a lower computational complexity, smaller detection error, and higher energy efficiency, which are validated through simulation.

In most of compressed sensing (CS) contributions, the sparsity of interesting signals is usually assumed to be prior. It may incur performance degradation of CS in such environments, where the sparsity order of channel occupancy changes dynamically. Therefore, sparsity order estimation (SOE) is vital, so that the sampling rate can be adjusted adaptively to exploit fully CS techniques. This paper studies the SOE problem for wideband cognitive radio. To investigate this issue, at first, the theoretical expression of sparsity order of the considered signals is derived via random matrix theory. The theoretical analysis shows that the sparsity order can be expressed with the ratio, which consists of the trace of measured signals' covariance matrix, compression sampling ratio, signal-to-noise ratio, and so on. And then, according to the theoretical expressions, a novel SOE approach is presented. In the proposed method, the trace of measured signals' covariance matrix is first calculated, and second, the sparsity order is estimated based on the trace. Compared with the previous work, the proposed algorithm, besides the lower computational complexity, has some superior performance, such as better robustness, smaller SOE error, and so on. Theoretical analysis and simulation results demonstrate the performance of the proposed approach.