Pre-trained language models have become a critical natural language processing component in many E-commerce applications. As businesses continue to evolve, the pre-trained models should be able to adopt new domain knowledge and new tasks. This paper proposes a novel sequential multi-task pre-trained language framework, ICL-BERT (In-loop Continual Learning BERT), which enables evolving the current model with new knowledge and new tasks. The contributions of ICL-BERT are (1) vocabularies and entities are optimized on E-commerce corpus; (2) a new glyph embedding is introduced to learn glyph information for vocabularies and entities; (3) specific and general tasks are designed to encode E-commerce knowledge for pre-training ICL-BERT; and (4) a new task-gating mechanism, called ICL (In-loop continual Learning), is proposed for sequential multi-task learning, which evolves the current model effectively and efficiently. Our evaluation results demonstrate that ICL-BERT outperforms existing models in both CLUE and e-commerce tasks, with an average accuracy improvement of 1.73% and 3.5%, respectively. Furthermore, ICL-BERT serves as a fundamental pre-trained language model that runs online in JingDong’s daily business.

Steady-state visual evoked potential-based brain-computer interfaces (SSVEP-BCIs) hold significant promise for enabling high-speed human-computer interaction in real-world scenarios. However, existing frequency-domain decoding methods treat frequency spectrum features (the real and imaginary spectrum features) as a single feature without considering their unique spatial and spectral characteristics, resulting in insufficient generalizable features and limited classification accuracy in cross-subject scenarios. To address this issue, we propose a Dual-Branch Attention-Based Frequency Domain Network (DB-AFDNet) to independently decode real and imaginary spectral components, aiming to acquire more discriminative and generalizable features for cross-subject applications. Specifically, we construct inter-branch attention similarity constraints to encourage the two branches to have similar attention properties, promoting to learn the consensus characteristics in the dual branches. Furthermore, we propose intra-branch orthogonality constraints to explore branch-specific discriminative features to learn generalizable features. Experimental studies on two public datasets, the Benchmark and Beta datasets, demonstrate that DB-AFDNet outperforms state-of-the-art methods in cross-subject classification, achieving a relative improvement of 1.36$\%$ and 1.45$\%$, respectively.

In this article, we formulate lifelong learning as an online transfer learning procedure over consecutive tasks, where learning a given task depends on the accumulated knowledge. We propose a novel theoretical principled framework, lifelong online learning, where the learning process for each task is in an incremental manner. Specifically, our framework is composed of two-level predictions: the prediction information that is solely from the current task; and the prediction from the knowledge base by previous tasks. Moreover, this article tackled several fundamental challenges: arbitrary or even non-stationary task generation process, an unknown number of instances in each task, and constructing an efficient accumulated knowledge base. Notably, we provide a provable bound of the proposed algorithm, which offers insights on the how the accumulated knowledge improves the predictions. Finally, empirical evaluations on both synthetic and real datasets validate the effectiveness of the proposed algorithm.

In this paper, we study two dimensional metric learning (2DML) for matrix data from both theoretical and algorithmic perspectives. We first investigate the generalization bounds of 2DML based on the notion of Rademacher complexity, which theoretically justifies the benefits of learning from matrices directly. Furthermore, we present a novel boosting-based algorithm that scales well with the feature dimension. Finally, we introduce an efficient rank-one correction algorithm, which is tailored to our boosting learning procedure to produce a low-rank solution to 2DML. As our algorithm works directly on the data in matrix representation, it scales well with the feature dimension, keeps the structure and dependence in the data, and has a more compact structure and much fewer parameters to optimize. Extensive evaluations on several benchmark data sets also empirically verify the effectiveness and efficiency of our algorithm.

Conversational Recommender Systems (CRSs) in E-commerce platforms aim to recommend items to users via multiple conversational interactions. Click-through rate (CTR) prediction models are commonly used for ranking candidate items. However, most CRSs are suffer from the problem of data scarcity and sparseness. To address this issue, we propose a novel knowledge-enhanced deep cross network (K-DCN), a two-step (pretrain and fine-tune) CTR prediction model to recommend items. We first construct a billion-scale conversation knowledge graph (CKG) from information about users, items and converations, and then pretrain CKG by introducing knowledge graph embedding method and graph convolution network to encode semantic and structural information respectively. To make the CTR prediction model sensible of current state of users and the relationship between dialogues and items, we introduce user-state and dialogue-interaction representations based on pre-trained CKG and propose K-DCN. In K-DCN, we fuse the user-state representation, dialogue-interaction representation and other normal feature representations via deep cross network, which will give the rank of candidate items to be recommended. We experimentally prove that our proposal significantly outperforms baselines and show it's real application in Alime.

In recent years, knowledge graphs have been widely applied to organize data in a uniform way and enhance many tasks that require knowledge, for example, online shopping which has greatly facilitated people's life. As a backbone for online shopping platforms, we built a billion-scale e-commerce product knowledge graph for various item knowledge services such as item recommendation. However, such knowledge services usually include tedious data selection and model design for knowledge infusion, which might bring inappropriate results. Thus, to avoid this problem, we propose a Pre-trained Knowledge Graph Model (PKGM) for our billion-scale e-commerce product knowledge graph, providing item knowledge services in a uniform way for embedding-based models without accessing triple data in the knowledge graph. Notably, PKGM could also complete knowledge graphs during servicing, thereby overcoming the common incompleteness issue in knowledge graphs. We test PKGM in three knowledge-related tasks including item classification, same item identification, and recommendation. Experimental results show PKGM successfully improves the performance of each task.

Learning from subject's calibration data can significantly improve the performance of a steady-state visually evoked potential (SSVEP)-based brain-computer interface (BCI), for example, the state-of-the-art target recognition methods utilize the learned subject-specific and stimulus-specific model parameters. Unfortunately, when dealing with new stimuli or new subjects, new calibration data must be acquired, thus requiring laborious calibration sessions, which becomes a major challenge in developing high-performance BCIs for real-life applications. This study investigates the feasibility of transferring the model parameters (i.e., the spatial filters and the SSVEP templates) across two different groups of visual stimuli in SSVEP-based BCIs. According to our exploration, we can extract a common spatial filter from the spatial filters across different stimulus frequencies and a common impulse response from the SSVEP templates across different neighboring stimulus frequencies, in which the common spatial filter is considered as the transferred spatial filter and the common impulse response is utilized to reconstruct the transferred SSVEP template according to the theory that an SSVEP is a superposition of the impulse responses. Then, we develop a transfer learning canonical correlation analysis (tlCCA) incorporating the transferred model parameters. For evaluation, we compare the recognition performance of the calibration-free, the calibration-based, and the proposed tlCCA on an SSVEP data set with 60 subjects. Experiment results prove that the spatial filters share commonality across different frequencies and the impulse responses share commonality across neighboring frequencies. More importantly, the tlCCA performs significantly better than the calibration-free algorithms, comparably to the calibration-based algorithm. Note to Practitioners-This work is motivated by the long calibration time problem in using an steady-state visually evoked potential (SSVEP)-based brain-computer interface (BCI) because most state-of-the-art frequency recognition methods consider merely the situation that the calibration data and the test data are from the same subject and the same visual stimulus. This article assumes that the model parameters share the stimulus-nonspecific knowledge in a limited stimulus frequency range, and thus, the subject's old calibration data can be reused to learn new model parameters for new visual stimuli. First, the model parameters can be decomposed into the stimulus-nonspecific knowledge (or subject-specific knowledge) and stimulus-specific knowledge. Second, the new model parameters can be generated via transferring the knowledge across stimulus frequencies. Then, a new recognition algorithm is developed using the transferred model parameters. Experiment results validate the assumptions, and moreover, the proposed scheme could be extended to other scenarios, such as when facing new subjects, or adopting new signal acquisition equipment, which would be helpful to the future development of zero-calibration SSVEP-based BCIs for real-life healthcare applications.

Common spatial pattern (CSP) is one of the most successful feature extraction algorithms for brain-computer interfaces (BCIs). It aims to find spatial filters that maximize the projected variance ratio between the covariance matrices of the multichannel electroencephalography (EEG) signals corresponding to two mental tasks, which can be formulated as a generalized eigenvalue problem (GEP). However, it is challenging in principle to impose additional regularization onto the CSP to obtain structural solutions (e.g., sparse CSP) due to the intrinsic nonconvexity and invariance property of GEPs. This article reformulates the CSP as a constrained minimization problem and establishes the equivalence of the reformulated and the original CSPs. An efficient algorithm is proposed to solve this optimization problem by alternately performing singular value decomposition (SVD) and least squares. Under this new formulation, various regularization techniques for linear regression can then be easily implemented to regularize the CSPs for different learning paradigms, such as the sparse CSP, the transfer CSP, and the multisubject CSP. Evaluations on three BCI competition datasets show that the regularized CSP algorithms outperform other baselines, especially for the high-dimensional small training set. The extensive results validate the efficiency and effectiveness of the proposed CSP formulation in different learning contexts.

Entity alignment (EA) aims to identify entities located in different knowledge graphs (KGs) that refer to the same real-world object. To learn the entity representations, most EA approaches rely on either translation-based methods which capture the local relation semantics of entities or graph convolutional networks (GCNs), which exploit the global KG structure. Afterward, the aligned entities are identified based on their distances. In this paper, we propose to jointly leverage the global KG structure and entity-specific relational triples for better entity alignment. Specifically, a global structure and local semantics preserving network is proposed to learn entity representations in a coarse-to-fine manner. Experiments on several real-world datasets show that our method significantly outperforms other entity alignment approaches and achieves the new state-of-the-art performance.

Data-driven classification approaches have substantially boosted the classification performance in steady-state visual evoked potentials (SSVEP)-based brain-computer interface (BCI). However, as a tradeoff to classification accuracy, a long calibration session is required to collect training data, which greatly reduces the applicability of BCI. In order to minimize the calibration effort while retaining good performance, this paper considers the problem of transferring knowledge from historical subjects to new subject, i.e., subject-independent SSVEP-based BCI. To tackle the problem, we propose a novel way to learn the transferable spatial filters by estimating the invariant task-related spatial filter subspace. The bases of the invariant subspace, which we call prototype spatial filters, are robust estimation of the task-related spatial filters. They can be generalized to the unseen subject for better recovering the latent signals. A new classification approach based on the prototype filters, namely transfer template and filter canonical correlation analysis (ttf-CCA), is then proposed and compared with the state-of-art approaches on the SSVEP benchmark data set. The feasibility of the proposed method is validated by the significant improvement on the classification accuracy and information transfer rate (ITR).

In this paper, a novel learning method called postboosting using extended G-mean (PBG) is proposed for online sequential multiclass imbalance learning (OS-MIL) in neural networks. PBG is effective due to three reasons. 1) Through postadjusting a classification boundary under extended G-mean, the challenging issue of imbalanced class distribution for sequentially arriving multiclass data can be effectively resolved. 2) A newly derived update rule for online sequential learning is proposed, which produces a high G-mean for current model and simultaneously possesses almost the same information of its previous models. 3) A dynamic adjustment mechanism provided by extended G-mean is valid to deal with the unresolved challenging dense-majority problem and two dynamic changing issues, namely, dynamic changing data scarcity (DCDS) and dynamic changing data diversity (DCDD). Compared to other OS-MIL methods, PBG is highly effective on resolving DCDS, while PBG is the only method to resolve dense-majority and DCDD. Furthermore, PBG can directly and effectively handle unscaled data stream. Experiments have been conducted for PBG and two popular OS-MIL methods for neural networks under massive binary and multiclass data sets. Through the analyses of experimental results, PBG is shown to outperform the other compared methods on all data sets in various aspects including the issues of data scarcity, dense-majority, DCDS, DCDD, and unscaled data.

Recently, multilayer extreme learning machine (ML-ELM) was applied to stacked autoencoder (SAE) for representation learning. In contrast to traditional SAE, the training time of ML-ELM is significantly reduced from hours to seconds with high accuracy. However, ML-ELM suffers from several drawbacks: 1) manual tuning on the number of hidden nodes in every layer is an uncertain factor to training time and generalization; 2) random projection of input weights and bias in every layer of ML-ELM leads to suboptimal model generalization; 3) the pseudoinverse solution for output weights in every layer incurs relatively large reconstruction error; and 4) the storage and execution time for transformation matrices in representation learning are proportional to the number of hidden layers. Inspired by kernel learning, a kernel version of ML-ELM is developed, namely, multilayer kernel ELM (ML-KELM), whose contributions are: 1) elimination of manual tuning on the number of hidden nodes in every layer; 2) no random projection mechanism so as to obtain optimal model generalization; 3) exact inverse solution for output weights is guaranteed under invertible kernel matrix, resulting to smaller reconstruction error; and 4) all transformation matrices are unified into two matrices only, so that storage can be reduced and may shorten model execution time. Benchmark data sets of different sizes have been employed for the evaluation of ML-KELM. Experimental results have verified the contributions of the proposed ML-KELM. The improvement in accuracy over benchmark data sets is up to 7%.

The adaptive Fourier decomposition (AFD) is a greedy iterative signal decomposition algorithm in the viewpoint of energy. Instead of using a fixed basis for decomposition, AFD uses an adaptive basis to achieve efficient energy extraction. In the conventional searching method, a new basis is searched from a large dictionary at every decomposition level. This usually results in a slow searching speed. To improve the efficiency, a fast searching method based on Nelder-Mead algorithm is proposed in this paper. The AFD with the proposed searching method is applied for electrocardiography (ECG) signals in which the selection ranges of four key parameters in the proposed searching method are determined based on simulation results of an artificial ECG signal. The simulation results of real ECG data shows that the computational time of the AFD based on the proposed searching method is just half of that based on the conventional searching method with similar reconstruction error.

Among the EEG-based BCIs, SSVEP-based BCIs have gained much attention due to the advantages of relatively high information transfer rate (ITR) and short calibration time. Although in SSVEP-based BCIs the frequency recognition methods using multiple channels EEG signals may provide better accuracy, using single channel would be preferable in a practical scenario since it can make the system simple and easy-to-use. To this goal, we propose a new single channel method based on wavelet-independent component analysis (WICA) in the SSVEP-based BCI, in which wavelet transform (WT) is applied to decompose a single channel signal into several wavelet components and then independent component analysis (ICA) is applied to separate the independent sources from the wavelet components. Experimental results show that most of the time the recognition accuracy of the proposed single channel method is higher than the conventional single channel method, power spectrum (PS) method.

The reduction of the muscle and electrode motion artifacts in ECG using the adaptive Fourier decomposition (AFD) is investigated. This is an extension of our previous work, in which AFD is first proposed for ECG denoising and its effectiveness in filtering out the additive Gaussian white noise is tested. This paper studies the AFD-based ECG denoising method for two types of ECG noise due to the electrode movement and the muscle contraction which are common and important in practice. In addition, some rules on the selection and adjustment of the AFD decomposition level are proposed. The tests on the MIT-BIH Arrhythmia Database indicate that this AFD-based denoising scheme performs better than the Butterworth lowpass filter, the wavelet transform and the empirical mode decomposition methods for ECG denoising with the muscle movement and electrode motion artifacts.

In this paper, we propose a modified visual stimulus generation method and feature detection algorithm to design a frequency and phase coding steady-state visual evoked potential (SSVEP) based brain-computer interface (BCI). By utilizing both frequency and phase information, we solve the harmonic frequency problem in our proposed SSVEP-BCI system. The offline experimental results show that the proposed feature detection algorithm can enhance the classification rate over 10% (from 69%±12% to 82%±8%) even though only one signal electrode is used and the harmonic frequencies (6.67Hz, 13.33Hz, 8.57Hz and 17.14Hz) are employed.