This paper 1 studies how to schedule hyperparameters to improve generalization of both centralized single-machine stochastic gradient descent (SGD) and distributed asynchronous SGD (ASGD). SGD augmented with momentum variants (e.g., heavy ball momentum (SHB) and Nesterov’s accelerated gradient (NAG)) has been the default optimizer for many tasks, in both centralized and distributed environments. However, many advanced momentum variants, despite empirical advantage over classical SHB/NAG, introduce extra hyperparameters to tune. The error-prone tuning is the main barrier for AutoML. Centralized SGD : We first focus on centralized single-machine SGD and show how to efficiently schedule the hyperparameters of a large class of momentum variants to improve generalization. We propose a unified framework called multistage quasi-hyperbolic momentum (Multistage QHM), which covers a large family of momentum variants as its special cases (e.g. vanilla SGD/SHB/NAG). Existing works mainly focus on only scheduling learning rate α ’s decay, while multistage QHM allows additional varying hyperparameters (e.g., momentum factor), and demonstrates better generalization than only tuning α . We show the convergence of multistage QHM for general nonconvex objectives. Distributed SGD : We then extend our theory to distributed asynchronous SGD (ASGD), in which a parameter server distributes data batches to several worker machines and updates parameters via aggregating batch gradients from workers. We quantify the asynchrony between different workers (i.e., gradient staleness), model the dynamics of asynchronous iterations via a stochastic differential equation (SDE), and then derive a PAC-Bayesian generalization bound for ASGD. As a byproduct, we show how a moderately large learning rate helps ASGD to generalize better. Our tuning strategies have rigorous justifications rather than a blind trial-and-error as we theoretically prove why our tuning strategies could decrease our derived generalization errors in both cases. Our strategies simplify the tuning process and beat competitive optimizers in test accuracy empirically. Our codes are publicly available https://github.com/jsycsjh/centralized-asynchronous-tuning.

Pre-trained concept representations are essential to many biomedical text mining and natural language processing tasks. As such, various representation learning approaches have been proposed in the literature. More recently, contextualized embedding approaches (i.e., BERT based models) that capture the implicit semantics of concepts at a granular level have significantly outperformed the conventional word embedding approaches (i.e., Word2Vec/GLoVE based models). Despite significant accuracy gains achieved, these approaches are often computationally expensive and memory inefficient. To address this issue, we propose a new representation learning approach that efficiently adapts the concept representations to the newly available data. Specifically, the proposed approach develops a knowledge-guided continual learning strategy wherein the accurate/stable context-information present in human-curated knowledge-bases is exploited to continually identify and retrain the representations of those concepts whose corpus-based context evolved coherently over time. Different from previous studies that mainly leverage the curated knowledge to improve the accuracy of embedding models, the proposed research explores the usefulness of semantic knowledge from the perspective of accelerating the training efficiency of embedding models. Comprehensive experiments under various efficiency constraints demonstrate that the proposed approach significantly improves the computational performance of biomedical word embedding models.

Stochastic gradient descent (SGD) augmented with various momentum variants (e.g. heavy ball momentum (SHB) and Nesterov's accelerated gradient (NAG)) has been the default optimizer for many learning tasks. Tuning the optimizer's hyperparameters is arguably the most time-consuming part of model training. Many new momentum variants, despite their empirical advantage over classical SHB/NAG, introduce even more hyperparameters to tune. Automating the tedious and error-prone tuning is essential for AutoML. This paper focuses on how to efficiently tune a large class of multistage momentum variants to improve generalization. We use the general formulation of quasi-hyperbolic momentum (QHM) and extend "constant and drop'', the widespread learning rate α scheduler where α is set large initially and then dropped every few epochs, to other hyperparameters (e.g. batch size b, momentum parameter β, instant discount factor ν). Multistage QHM is a unified framework which covers a large family of momentum variants as its special cases (e.g. vanilla SGD/SHB/NAG). Existing works mainly focus on scheduling α's decay, while multistage QHM allows additional varying hyperparameters such as b, β, and ν, and demonstrates better generalization ability than only tuning α. Our tuning strategies have rigorous justifications rather than a blind trial-and-error. We theoretically prove why our tuning strategies could improve generalization. We also show the convergence of multistage QHM for general nonconvex objective functions. Our strategies simplify the tuning process and beat competitive optimizers in test accuracy empirically.

Indexing biomedical research articles with Medical Subject Headings (MeSH) can greatly facilitate biomedical research and information retrieval. Currently MeSH indexing is performed by human experts. To alleviate the time consumption and monetary cost caused by manual indexing, many automatic MeSH indexing models have been developed, such as MeSHProbeNet, DeepMeSH, and NLM’s official model Medical Text Indexer. In this article, we propose an end-to-end framework, MeSHProbeNet-P, which extends MeSHProbeNet with personalizable MeSH probes. In MeSHProbeNet-P, each MeSH probe carries certain aspects of biomedical knowledge and extracts related information from input articles. MeSHProbeNet-P is able to automatically personalize its MeSH probes for different input articles to ensure that the current MeSH probes best fit the current input article and the most informative features can be extracted from the article. We demonstrate the effectiveness of MeSHProbeNet-P in a real-world large-scale MeSH indexing challenge. MeSHProbeNet-P won the first place in the first batch of Task A in the 2019 BioASQ challenge. The result on the first test set of the challenge is reported in this article. We also provide ablation studies to show the advantages of personalizable MeSH probes.

The dynamic network data have become ubiquitous with the rapid development of Internet and smart devices. To effectively manage the involved vertices in networks, it is crucial to track the special community patterns and analyze the relationships among vertices. In this paper, we propose a new method to measure the coherence strength, also referred to as community consistency, of a community over a specific observation period. The measurement of community consistency is especially challenging given the dynamic community structure over time, i.e., vertices can leave their original communities and join new communities. In order to interpret the causes of evolving community structure and model the influence of evolving community structure on community consistency, we introduce an influence propagation process having a causal relation with the community consistency. Specifically, a generative model is proposed to combine the influence propagation and the network topological structure at each time step. The proposed influence-based approach for modeling evolution can be instantiated in a variety of real-world network data. The comprehensive experiments on both synthetic and real-world datasets demonstrate the superiority of the proposed framework in estimating the community consistency. Besides, we conduct a case study to show the effectiveness of the proposed method in real-world applications.

Missing data, which commonly appears in multivariate time series, has been widely recognized as a key challenge in time series analysis. Many commonly used imputation methods either ignore the temporal dependencies of time series data, or do not adequately utilize the relationships among variables. State-ofthe-art methods on time series imputation are built on Recurrent Neural Networks (RNNs), which utilize the historical information to estimate current values sequentially. However, RNNs rely heavily on the output of nearby timestamps, which may lead to important information lost for long sequences. Moreover, individual variables typically present different dynamics and missingness patterns, which is neglected by the global RNN hidden states. In this paper, we propose an imputation framework to learn both global and local dependencies of multivariate time series, as well as a multi-dimensional self-attention to learn capture distant correlations across both time and feature. Extensive experiments show that the proposed framework outperforms the state-of-the-art methods in the imputation task, and benefits the downstream task.

This paper develops the Correlation Networks (CorNet) architecture for the extreme multi-label text classification (XMTC) task, where the objective is to tag an input text sequence with the most relevant subset of labels from an extremely large label set. XMTC can be found in many real-world applications, such as document tagging and product annotation. Recently, deep learning models have achieved outstanding performances in XMTC tasks. However, these deep XMTC models ignore the useful correlation information among different labels. CorNet addresses this limitation by adding an extra CorNet module at the prediction layer of a deep model, which is able to learn label correlations, enhance raw label predictions with correlation knowledge and output augmented label predictions. We show that CorNet can be easily integrated with deep XMTC models and generalize effectively across different datasets. We further demonstrate that CorNet can bring significant improvements over the existing deep XMTC models in terms of both performance and convergence rate. The models and datasets are available at: https://github.com/XunGuangxu/CorNet.

Hypothesis generation (HG) refers to the task of mining meaningful implicit association between unlinked biomedical concepts. The majority of prior studies have focused on uncovering these implicit linkages from static snapshots of the corpus, thereby largely ignoring the temporal dynamics of medical concepts. More recently, a few initial studies attempted to overcome this issue by modelling the temporal change of concepts from natural language text. However, they still fail to leverage the evolutionary features of concepts from contemporary knowledge-bases (KB's) such as semantic lexicons and ontologies. In practice such KB's contain up-to-date information that is important to incorporate, especially, in highly evolving domains such as biomedicine. Furthermore, considering the complementary strength of these sources of information - corpus and ontology - a few natural questions arise: Can joint modelling of (co)-evolutionary dynamics from these resources aid in encoding the temporal features at a granular level? Can the mutual evolution between these intertwined resources lead to better predictive effects? To answer these questions, in this study, we present a novel HG framework that unearths the latent associations between concepts by modeling their co-evolution across complementary sources of information. More specifically, the proposed approach adopts a shared temporal matrix factorization framework that models the co-evolution of concepts across both corpus and KB. Extensive experiments on the largest available biomedical corpus validates the effectiveness of the proposed approach.

Recent advances in unsupervised language processing methods have created an opportunity to exploit massive text corpora for developing high-quality vector space representation (also known as word embeddings) of words. Towards this direction, practitioners have developed and applied several data driven embedding models with quite good rate of success. However, a drawback of these models lies in their premise of static context; wherein, the meaning of a word is assumed to remain the same over the period of time. This is limiting because it is known that the semantic meaning of a concept evolves over time. While such semantic drifts are routinely observed in almost all the domains; their effect is acute in domain such as biomedicine, where the semantic meaning of a concept changes relatively fast. To address this, in this study, we aim to learn temporally aware vector representation of medical concepts from the timestamped text data, and in doing so provide a systematic approach to formalize the problem. More specifically, a dynamic word embedding based model that jointly learns the temporal characteristics of medical concepts and performs across time-alignment is proposed. Apart from capturing the evolutionary characteristics in an optimal manner, the model also factors in the implicit medical properties useful for a variety of bio-medical applications. Empirical studies conducted on two important bio-medical use cases validates the effectiveness of the proposed approach and suggests that the model not only learns quality embeddings but also facilitates intuitive trajectory visualizations.

Word embeddings are finding their increasing application in a variety of biomedical Natural Language Processing (bioNLP) tasks, ranging from drug discovery to automated disease diagnosis. While these word embeddings in their entirety have shown meaningful syntactic and semantic regularities, however, the meaning of individual dimensions remains elusive. This becomes problematic both in general and particularly in sensitive domains such as bio-medicine, wherein, the interpretability of results is crucial to its widespread adoption. To address this issue, in this study, we aim to improve the interpretability of pre-trained word embeddings generated from a text corpora, and in doing so provide a systematic approach to formalize the problem. More specifically, we exploit the rich categorical knowledge present in the biomedical domain, and propose to learn a transformation matrix that transforms the input embeddings to a new space where they are both interpretable and retain their original expressive features. Experiments conducted on the largest available biomedical corpus suggests that the model is capable of performing interpretability that resembles closely to the human-level intuition.

Recent advances in attention networks have gained enormous interest in time series data mining. Various attention mechanisms are proposed to soft-select relevant timestamps from temporal data by assigning learnable attention scores. However, many real-world tasks involve complex multivariate time series that continuously measure target from multiple views. Different views may provide information of different levels of quality varied over time, and thus should be assigned with different attention scores as well. Unfortunately, the existing attention-based architectures cannot be directly used to jointly learn the attention scores in both time and view domains, due to the data structure complexity. Towards this end, we propose a novel multi-view attention network, namely MuVAN, to learn fine-grained attentional representations from multivariate temporal data. MuVAN is a unified deep learning model that can jointly calculate the two-dimensional attention scores to estimate the quality of information contributed by each view within different timestamps. By constructing a hybrid focus procedure, we are able to bring more diversity to attention, in order to fully utilize the multi-view information. To evaluate the performance of our model, we carry out experiments on three real-world benchmark datasets. Experimental results show that the proposed MuVAN model outperforms the state-of-the-art deep representation approaches in different real-world tasks. Analytical results through a case study demonstrate that MuVAN can discover discriminative and meaningful attention scores across views over time, which improves the feature representation of multivariate temporal data.

Given two topics of interest (A, C) that are otherwise disconnected - for instance two concepts: a disease ("Migraine") and a therapeutic substance ("Magnesium") - this paper attempts to find the conceptual bridges (e.g., serotonin (B)) that connects them in a novel way. This problem of mining implicit linkage is known as hypotheses generation and its potential to accelerate scientific progress is widely recognized. Almost all of the prior studies to tackle this problem ignore the temporal dynamics of concepts. This is limiting because it is known that the semantic meaning of a concept evolves over time. To overcome this issue, in this study, we define this problem as mining time-aware Top-k conceptual bridges, and in doing so provide a systematic approach to formalize the problem. Specifically, the proposed model first extracts relevant entities from the corpus, represents them in time-specific latent spaces, and then further reasons upon it to generate novel and experimentally testable hypotheses. The key challenge in this approach is to learn a mapping function that encodes the temporal characteristics of concepts and aligns the across-time latent spaces. To solve this, we propose an effective algorithm that learns precise mapping sensitive to both global and local semantics of the input query. Both qualitative and quantitative evaluations performed on the largest available biomedical corpus substantiate the importance of leveraging the evolutionary semantics of medical concepts and suggests that the generated hypotheses are novel and worthy of clinical trials.

As news reading on social media becomes more and more popular, fake news becomes a major issue concerning the public and government. The fake news can take advantage of multimedia content to mislead readers and get dissemination, which can cause negative effects or even manipulate the public events. One of the unique challenges for fake news detection on social media is how to identify fake news on newly emerged events. Unfortunately, most of the existing approaches can hardly handle this challenge, since they tend to learn event-specific features that can not be transferred to unseen events. In order to address this issue, we propose an end-to-end framework named Event Adversarial Neural Network (EANN), which can derive event-invariant features and thus benefit the detection of fake news on newly arrived events. It consists of three main components: the multi-modal feature extractor, the fake news detector, and the event discriminator. The multi-modal feature extractor is responsible for extracting the textual and visual features from posts. It cooperates with the fake news detector to learn the discriminable representation for the detection of fake news. The role of event discriminator is to remove the event-specific features and keep shared features among events. Extensive experiments are conducted on multimedia datasets collected from Weibo and Twitter. The experimental results show our proposed EANN model can outperform the state-of-the-art methods, and learn transferable feature representations.

The technological advancements in biomedical domain has led to a tremendous growth of unstructured data; primarily a result of increased publication of findings. At the same time, a corresponding interest in the Natural Language Processing (NLP) community to develop scalable methodologies to exploit such massive unlabeled corpora for unsupervised language processing has resulted in new opportunities towards developing semantic sensitive models. Amongst them, the field of word embeddings has garnered significant attention due to its capability to understand implicit semantics. However such data driven models are largely agnostic of the rich explicit semantic knowledge available in the biomedical domain in the form of vocabularies and ontologies. This is problematic because it leads to a poor representation of words with little local context and its effect is acute in biomedical domain. In this paper, we propose a novel model (MeSH2Vec) that jointly exploits both contextual information and available explicit semantic knowledge to learn externally augmented word embeddings. Unlike existing approaches, the proposed methodology is more dexterous in its ability to handle relationships between indirectly related concepts. The 13% improvement in the correlation to experts, shown on experiments involving biomedical concept similarity and relatedness task validates the effectiveness of the proposed approach and demonstrates the importance of incorporating human curated knowledge in the process of generating word embeddings.

Literature based discovery (LBD) is a task that aims to uncover hidden associations between non-interacting scientific concepts by rationally connecting independent nuggets of information. Broadly, prior approaches to LBD include use of: a) distributional statistics and explicit representation, b) graph-theoretic measures, and c) supervised machine learning methods to find associations. However, purely distributional approaches may not necessarily entail semantically meaningful association and graph-theoretic approaches suffer from scalability issues. While supervised machine learning based approaches have the potential to elucidate associations, the training data required is too expensive to generate. In this paper we propose a novel dynamic Medical Subject Heading (MeSH) embedding model which is able to model the evolutionary behavior of medical concepts to uncover latent associations between them. The proposed model allows us to learn the evolutionary trajectories of MeSH embeddings and detect informative terms. Hence, based on the dynamic MeSH embeddings, meaningful medical hypotheses can be efficiently generated. To evaluate the efficacy of the proposed model, we perform both qualitative and quantitative evaluation. The results demonstrate that leveraging the evolutionary features of MeSH concepts is an effective way for predicting novel associations.

Time series data mining has gained increasing attention in health domain. Recently, researchers attempt to employ Natural Language Processing (NLP) to health data mining, in order to learn proper representations of discrete medical concepts from Electronic Health Records (EHRs). However, existing models do not take continuous physiological records into account, which are naturally existed in EHRs. The major challenges for this task are to model non-obvious representations from observed high dimensional biosignals, and to interpret the learned features. To address these issues, we propose Wave2Vec, an end-to-end deep learning model, to bridge the gap between biosignal processing and language modeling. Wave2Vec jointly learns both inherent and embedding representations of biosignals at the same time. To evaluate the performance of our model in clinical task, we carry out experiments on two real world benchmark biosignal datasets. Experimental results show that the proposed Wave2Vec model outperforms the six feature leaning baselines in biosignal processing.

A text corpus typically contains two types of context information -- global context and local context. Global context carries topical information which can be utilized by topic models to discover topic structures from the text corpus, while local context can train word embeddings to capture semantic regularities reflected in the text corpus. This encourages us to exploit the useful information in both the global and the local context information. In this paper, we propose a unified language model based on matrix factorization techniques which 1) takes the complementary global and local context information into consideration simultaneously, and 2) models topics and learns word embeddings collaboratively. We empirically show that by incorporating both global and local context, this collaborative model can not only significantly improve the performance of topic discovery over the baseline topic models, but also learn better word embeddings than the baseline word embedding models. We also provide qualitative analysis that explains how the cooperation of global and local context information can result in better topic structures and word embeddings.

Learning semantics based on context information has been researched in many research areas for decades. Context information can not only be directly used as the input data, but also sometimes used as auxiliary knowledge to improve existing models. This survey aims at providing a structured and comprehensive overview of the research on context learning. We summarize and group the existing literature into four categories, Explicit Analysis, Implicit Analysis, Neural Network Models, and Composite Models, based on the underlying techniques adopted by them. For each category, we talk about the basic idea and techniques, and also introduce how context information is utilized as the model input or incorporated into the model to enhance the performance or extend the domain of application as auxiliary knowledge. In addition, we discuss the advantages and disadvantages of each model from both the technical and practical point of view.

The development of Internet and social networks has provided more emerging network data which facilitates the dynamic network analysis. In this paper, we propose a new method to measure coherence strength, also referred to as community consistency, of a community under dynamic settings. In order to better interpret the influence of evolving community structure on community consistency, we model the problem as one of influence propagation processes having a causal relation with the community consistency. To this effect a generative model is proposed to combine the influence propagation and the network topological structure at each time stamp. Our comprehensive experiments on both synthetic and real-world datasets demonstrate the superiority of the proposed framework in estimating the community consistency.

The development of social networks has not only improved the online experience, but also stimulated the advances in knowledge mining so as to assist people in planning their offline social events. Users can explore their favorite events, such as celebrations and symposiums, through the pictures and the posts from their friends on social networks. An effective event recommendation can offer great convenience for both event organizers and participants, which yet remains extremely challenging due to a wide range of practical concerns. In this paper we propose a novel recommendation framework, which combines the information from multiple sources and establishes a connection between the online knowledge and the event participation.

Discovering topics in short texts, such as news titles and tweets, has become an important task for many content analysis applications. However, due to the lack of rich context information in short texts, the performance of conventional topic models on short texts is usually unsatisfying. In this paper, we propose a novel topic model for short text corpus using word embeddings. Continuous space word embeddings, which is proven effective at capturing regularities in language, is incorporated into our model to provide additional semantics. Thus we model each short document as a Gaussian topic over word embeddings in the vector space. In addition, considering that background words in a short text are usually not semantically related, we introduce a discrete background mode over word types to complement the continuous Gaussian topics. We evaluate our model on news titles from data sources like abcnews, showing that our model is able to extract more coherent topics from short texts compared with the baseline methods and learn better topic representation for each short document.

The increasing popularity of mobile devices has brought severe challenges to device usability and big data analysis. In this paper we investigate the intellectual recommender system on cell phones by incorporating mobile data analysis. Nowadays with the development of smart phones, more and more applications have emerged on various areas, such as entertainment, education and health care. While these applications have brought great convenience to people's daily life, they also provide tremendous opportunities for analyzing users' interests. In this work we develop an Android background service to collect the user behaviors and analyze their preferences based on their Android application usage. As one of the most intuitive media for visual representation, videos with various types of contents are recommended to users based on a proposed graphical model. The proposed model jointly utilizes the textual descriptions of Android applications and videos, as well as the extracted video content based features. Besides, by analyzing the user's habit of application usage we seamlessly integrate the user's personal interests during the recommendation. The extensive comparisons to multiple baselines reveal the superiority of the proposed model on the recommendation quality. Furthermore, we conduct experiments on personalized recommendation to demonstrate the capacity of the proposed model in effectively analyzing the user's personal interests.

Morden physiological analysis begins to involve more and more types of information. Electroencephalogram (EEG) signals as a typical example is starting to be analyzed with facial expressions videos to detect emotions. Emotions play an important role in the daily life of human beings, the need and importance of automatic emotion recognition has grown with increasing role of human computer interface applications. In this paper, we concentrate on recognition of the emotions jointly from "inner" and "outer" reactions, which are electroencephalogram (EEG) signals and facial expression video. Due to the streaming nature of this problem, the data volume and velocity is very challenging. We address these challenges from the theoretic perspective and propose a real time algorithm based on EEG signals and synchronized facial video to learn feature vector jointly. Our algorithm consists of an unsupervisedly EEG dictionary component based on deep learning theorem, and a probability pooling component transforms a continuous sequential signal into an EEG "sentence" which consists of a sequence of EEG words. The EEG sentence is then jointly learned with video features into a new fixed length feature representation for emotion classification. We overcome several computational challenges on the data based on the idea of convolution and pooling, and we conduct extensive evaluation for each component of our model. We also demonstrate the state-of-the-art classification result on real-world dataset. The superior performances on the emotion recognition task indicates that 1) the natural language scenario can be applied in EEG sequences and 2) borrowing video modality can increase the overall performance.