Continual pretraining enables Large Language Models (LLMs) to adapt to specialized domains like medicine and law.However, we observe a consistent phenomenon across different model sizes and domains: a temporary performance drop at the start of the continual pretraining process, followed by a performance recovery phase.To gain a deeper understanding of this issue, we use the stability gap-a concept adapted from the visual domain-which explains this initial drop arises from instability in the model's general abilities.We validate this hypothesis through a series of experiments.To address this initial instability and enhance LLM performance within a fixed compute budget, we propose a training strategy that mitigates instability by increasing the number of epochs, alongside two data sampling strategies targeting data domain relevance and corpus distribution.We conduct experiments on Llamafamily models to validate the effectiveness of our strategies for continual pretraining and instruction tuning in medical and legal domains.Our strategies improve the average medical task performance of the OpenLlama-3B model from 36.2% to 40.7% using only 40% of the original training budget, while also enhancing general task performance without causing forgetting.Furthermore, we apply our strategies to continually pre-train and instruction-tune the Llama-3-8B model.The resulting model, Llama-3-Physician 1 , achieves the best medical performance among open-source models and rivals GPT-4 on specific tasks.

Class incremental learning (CIL) is a challenging setting of continual learning, which learns a series of tasks sequentially. Each task consists of a set of unique classes. The key feature of CIL is that no task identifier (or task-id) is provided at test time. Predicting the task-id for each test sample is a challenging problem. An emerging theory-guided approach (called TIL+OOD) is to train a task-specific model for each task in a shared network for all tasks based on a task-incremental learning (TIL) method to deal with catastrophic forgetting. The model for each task is an out-of-distribution (OOD) detector rather than a conventional classifier. The OOD detector can perform both within-task (in-distribution (IND)) class prediction and OOD detection. The OOD detection capability is the key to task-id prediction during inference. However, this paper argues that using a traditional OOD detector for task-id prediction is sub-optimal because additional information (e.g., the replay data and the learned tasks) available in CIL can be exploited to design a better and principled method for task-id prediction. We call the new method TPL (Task-id Prediction based on Likelihood Ratio). TPL markedly outperforms strong CIL baselines and has negligible catastrophic forgetting. The code of TPL is publicly available at https://github.com/linhaowei1/TPL.

Existing research has shown that a multilingual pre-trained language model fine-tuned with one (source) language also performs well on downstream tasks for non-source languages, even though no fine-tuning is done on these languages.However, there is a clear gap between the performance of the source language and that of the non-source languages.This paper analyzes the fine-tuning process, discovers when the performance gap changes and identifies which network weights affect the overall performance most.Additionally, the paper seeks to answer to what extent the gap can be reduced by reducing forgetting.Based on the analysis results, a method named Fine-tuning slow and fast with four training policies is proposed to address these issues.Experimental results show the proposed method outperforms baselines by a clear margin.

Existing continual learning (CL) research regards catastrophic forgetting (CF) as almost the only challenge. This paper argues for another challenge in class-incremental learning (CIL), which we call cross-task class discrimination (CTCD), i.e., how to establish decision boundaries between the classes of the new task and old tasks with no (or limited) access to the old task data. CTCD is implicitly and partially dealt with by replay-based methods. A replay method saves a small amount of data (replay data) from previous tasks. When a batch of current task data arrives, the system jointly trains the new data and some sampled replay data. The replay data enables the system to partially learn the decision boundaries between the new classes and the old classes as the amount of the saved data is small. However, this paper argues that the replay approach also has a dynamic training bias issue which reduces the effectiveness of the replay data in solving the CTCD problem. A novel optimization objective with a gradient-based adaptive method is proposed to dynamically deal with the problem in the online CL process. Experimental results show that the new method achieves much better results in online CL.

Although deep neural networks (DNNs) have achieved good results on some common datasets of synthetic aperture radar (SAR) automatic target recognition (ATR), DNNs are opaque and difficult to interpret, which limits its practical application. In view of this, many interpretable models have been proposed in recent years. However, the previous interpretable models only represent transparent network structures and cannot be called real “interpretable” models. We think that the interpretability of a model contains two meanings: on the one hand, the decision process of the model is transparent, and on the other hand, the decision-making basis of the model should be reasonable and conform to human cognition. For this reason, we propose a new SAR image recognition model based on adversarial defense, namely SAR-AD-BagNet. Not only does it have a transparent decision-making process, but it also has a more reasonable basis for decision-making. In addition, the model also has high SAR image recognition accuracy and strong adversarial robustness.

Catastrophic forgetting is a key obstacle to continual learning. One of the state-of-the-art approaches is orthogonal projection. The idea of this approach is to learn each task by updating the network parameters or weights only in the direction orthogonal to the subspace spanned by all previous task inputs. This ensures no interference with tasks that have been learned. The system OWM that uses the idea performs very well against other state-of-the-art systems. In this paper, we first discuss an issue that we discovered in the mathematical derivation of this approach and then propose a novel method, called AOP (Adaptive Orthogonal Projection), to resolve it, which results in significant accuracy gains in empirical evaluations in both the batch and online continual learning settings without saving any previous training data as in replay-based methods.

Time-of-flight (ToF) of ultrasonic waves is essential for petroleum well logging to draw borehole-wall images. This paper proposed a method that boosted accuracy of ToFs estimation in a complex geological environment. Unlike other classical methods, the proposed one adopts a one-dimensional convolutional neural network (1D-CNN) as a backbone to extract latent information related to ToF. For handling the shortage of ultrasonic waves with annotated ToFs, theoretical ultrasonic waves generated by manifold mathematical models are utilized as source domain to train the model, which is applied to estimate practical ultrasonic ToFs. Furthermore, since the distribution divergences exist between theoretical ultrasonic models and practical ones, Maximum Mean Discrepancy (MMD) and CORrelation ALignment (CORAL) as discrepancy loss functions are used to evaluate the distribution divergences between two domain datasets and to optimize the entire model. Tests on the ultrasonic waves acquired by an experimental well logging device demonstrate the proposed method has satisfactory performances.

This paper proposed a new online continual learning approach called OCM based on mutual information (MI) maximization. It achieves two objectives that are critical in dealing with catastrophic forgetting (CF). (1) It reduces feature bias caused by cross entropy (CE) as CE learns only discriminative features for each task, but these features may not be discriminative for another task. To learn a new task well, the network parameters learned before have to be modified, which causes CF. The new approach encourages the learning of each task to make use of the full features of the task training data. (2) It encourages preservation of the previously learned knowledge when training a new batch of incrementally arriving data. Empirical evaluation shows that OCM substantially outperforms the latest online CL baselines. For example, for CIFAR10, OCM improves the accuracy of the best baseline by 13.1% from 64.1% (baseline) to 77.2% (OCM).The code is publicly available at https://github.com/gydpku/OCM.

Microwave imaging with large rotating angle and sparse sampling is an attractive approach to obtain the high-resolution target image with reduced radar resource. However, the popular imaging methods, e.g., Range-Doppler (RD), back projection (BP), and sparse recovery (SR), are difficult to deal with large rotating angle and sparse sampling simultaneously. In recent years, deep learning (DL) has been widely studied and been successfully used to handle the problems in computer vision. However, since most existing DL networks are put forward for the real visual image and a large amount of data is essential for network training, DL cannot be directly used to process the complex and sparse target echo for microwave imaging. In this article, a new learning imaging framework is proposed and a model-driven learning imaging network (MDLI-Net) is built for high-resolution microwave imaging with large rotating angle and sparse sampling. In the proposed framework, the electromagnetic scattering model is used to generate the training data efficiently, and the sparse microwave imaging theory is applied to guide the design of the deep imaging network. By inputting the 2-D sparse complex-valued target echo, the trained MDLI-Net can output the high-resolution and focused target image efficiently. The effectiveness of the proposed learning imaging method is validated by experiment results with both simulated and real data.

Multiple-input multiple-output (MIMO) radar imaging is a new technique to obtain the radar image of aerospace targets. Orthogonal waveform design is one of the important issues for MIMO radar imaging. However, the fully orthogonal waveforms in the same frequency and with the arbitrary time delay do not exist in practice. Thus, the imaging result using nonorthogonal waveforms based on matched filtering (MF) method is usually unsatisfactory if further processing like digital beam forming (DBF) is not used. Sparse recovery (SR) method is possible to restrain the mutual interference of nonorthogonal waveforms by exploiting the sparsity of targets and improve the imaging quality. In this article, waveform design issue in SR-based MIMO imaging method is studied. The difference in the designs of waveforms in MF method and SR method is discussed. Based on requirements analysis, a comprehensive optimization model is built for waveform design and the existing cycle algorithm (CA) is modified to solve the model. Considering the fact that the target scene is always changing, waveforms should be adjusted along with the dynamic scene. Therefore, an adaptive waveform optimization method is further proposed based on the cognition of target scene. The dimension of SR model is reduced and the waveforms are optimized according to the cognitive target length. Moreover, based on the reconstructed target range profiles, transmitting waveforms together with recovery algorithm are further optimized to match the target better. Simulation results show that the waveforms after optimization are better than the nonoptimized waveforms and the proposed adaptive optimization method is valid and robust for the dynamic target scene.

The performance of the angle estimation algorithm based on the two-order or higher order cumulants in the impact noise background will decline sharply. Therefore, it is necessary to study the new algorithm to estimate target angle in the impact noise background. In order to solve the angle estimation problem of coherent sources in the impulse noise background, a conjugate rotation invariant subspace algorithm based on reduced order fractional lower order covariance matrix is proposed. Use the reduced dimension lower order fraction covariance matrix to reduce the impulse noise influence. And according to the conjugate rotation invariant subspace, the coherent source is decohered. The Monte-Carlo experiments show that the proposed algorithm has the advantages of high estimation probability and low root mean square error in the case of low signal-to-noise ratio, compared with the existing FLOM-MUSIC algorithm and FLOM-Unitary ESPRIT algorithm.

Multiple‐input‐multiple‐output (MIMO) radar is attractive in moving targets imaging, which is able to solve the difficulty of complex motion compensation. In this article, a three‐dimensional imaging method using bistatic MIMO radar is proposed. Compared with monostatic MIMO radar, the bistatic system can provide the complementary information, and the imaging process is not the same as the monostatic case. Furthermore, considering the image sparsity and the spatial limitation of radar targets, a fast smoothed L0 norm algorithm is proposed to achieve the high resolution in cross‐range directions with limited antennas. The experimental results demonstrate the validity and efficiency of the proposed method.

Space-time adaptive processing (STAP) based on sparse recovery (SR-STAP) techniques exhibits significantly better performance than conventional STAP algorithms within a very small number of snapshots. However, when the clutter patches do not locate exactly on the discrete space-time grid points, the performances of SR-STAP algorithms degrade severely. In this letter, a low-complexity off-grid STAP algorithm based on local search clutter subspace estimation is proposed to overcome this issue. In the proposed algorithm, the global atoms are first selected from the reduced-dimension global STAP dictionary using the design selection criterion. Then, the optimal atoms are searched from the local STAP dictionary. Finally, these space-time steering vectors corresponding to the optimal atoms are used to construct the clutter subspace iteratively, and the STAP weight is obtained by projecting the snapshot on the subspace orthogonal to the clutter subspace. Numerical experiments with both simulated and Mountain-Top data are carried out to demonstrate the effectiveness of the proposed algorithm.