Proteins are central to biological processes and indispensable for living organisms. Accurate representation of proteins is crucial, especially in drug development. Recent advances have applied machine learning for unsupervised protein representation learning. However, these approaches often focus solely on the amino acid sequence of proteins and lack factual knowledge about proteins and their interactions, thus limiting their performance. In this study, we present GOProteinGNN, a novel architecture that enhances protein language models by integrating protein knowledge graph information during the creation of amino acid level representations. Our approach allows for the integration of information at both the individual amino acid level and the entire protein level, enabling a comprehensive and effective learning process through graph-based learning. By doing so, we can capture complex relationships and dependencies between proteins and their functional annotations, resulting in more robust and contextually enriched protein representations. Unlike previous methods, GOProteinGNN uniquely learns the entire protein knowledge graph during training, which allows it to capture broader relational nuances and dependencies beyond mere triplets as done in previous work. We perform a comprehensive evaluation on several downstream tasks, demonstrating that GOProteinGNN consistently outperforms previous methods, showcasing its effectiveness and establishing it as a state-of-the-art solution for protein representation learning. We discuss the practical integration of GOProteinGNN in a laboratory setting for lipid nanoparticle-based drug delivery, aiming to bypass the blood-brain barrier and discover novel components, with positive results observed in mice.

Within the realm of drug development, the transition from successful animal trials to human clinical efficacy remains a daunting challenge. While initial outcomes may appear promising in animal studies, ensuring similar effectiveness in humans, especially across specific target tissues, presents a significant obstacle. To address this pressing concern, we introduce a novel generative model tailored to optimize molecules that have demonstrated efficacy in rats for enhanced performance in specific human tissues. Central to our solution is the transformer architecture, enhanced with intricate mechanisms such as molecule self-attention within the encoder and a novel dedicated tissue-specific generator. Intuitively, by learning to generate molecules simultaneously from multiple tissues, the generative model enhances its ability to perform the necessary adaptations from rats to humans. Through rigorous empirical evaluation across various tissues, our model consistently exhibits remarkable efficacy compared to existing methods. We anticipate that this model has the potential to minimize the requirement for lengthy and inconclusive trials, thereby streamlining the drug development process.

Dynamic temporal graphs evolve over time, adding and removing nodes and edges between time snapshots. The tasks performed on such graphs are diverse and include detecting temporal trends, finding graph-to-graph similarities, and graph visualization and clustering. For all these tasks, it is necessary to embed the entire graph in a low-dimensional space by using graph-level representations instead of the more common node-level representations. This embedding requires handling the appearance of new nodes over time as well as capturing temporal patterns of the entire graph. Most existing methods perform temporal node embeddings and focus on different methods of aggregating them for a graph-based representation. In this work, we propose an end-to-end architecture that captures both the node embeddings and their influence in a structural context during a specific time period of the graph. We present GraphERT (Graph Embedding Representation using Transformers), a novel approach to temporal graph-level embeddings. Our method pioneers the use of Transformers to seamlessly integrate graph structure learning with temporal analysis. By employing a masked language model on sequences of graph random walks, together with a novel temporal classification task, our model not only comprehends the intricate graph dynamics but also unravels the temporal significance of each node and path. This novel training paradigm empowers GraphERT to capture the essence of both the structural and temporal aspects of graphs, surpassing state-of-the-art approaches across multiple tasks on real-world datasets.

Drug development is a long and costly process consisting of several stages that can take many years to complete. One of the early stage's goals is to optimize a novel chemical compound to be active against a target protein associated with the disease. Often machine learning techniques are used to improve the procedure of discovering and optimizing potential drug candidates. The goal of molecule optimization is, given an input molecule, to produce a new molecule that is chemically similar to the input molecule but with an improved property. We present a novel algorithm that during optimization divides a molecule into two disjoint substructures that we call: the molecule chains and the molecule core. Our approach is inspired by expert design of chemical compounds that employ a fundamental molecular template and add to it chemical functional groups to generate compounds with desired properties. We train a model to generate the molecule chains with the desired properties for optimization, which are then attached to the molecule core to construct a novel molecule with high similarity to the input molecule. This is achieved by selective masking of pairs of input molecules' chains and cores during training. Additionally, we demonstrate the extension of this approach to data-scarce tasks, like targeting a drug to a novel protein. We first evaluate our method on standard molecule optimization tasks such as inhibition against glycogen synthase kinase-3 beta (GSK3β). We then empirically compared the model performance with the state-of-the-art algorithms over 21 novel proteins and show superior performance.

Molecular optimization seeks to improve a given molecule's therapeutic profile. It is a key challenge in drug development, but it is difficult due to the constraints of molecular similarity to the original molecule and the size of the chemical space to explore. Numerous works tackled this problem with initial success. Unlike previous works that focus on generating molecules that optimize chemical properties, we focus on the optimization while attempting to "move away" from patented molecules. We present a novel loss function and its utilization in numerous types of molecular optimization algorithms. The loss allows to improve molecular properties while decreasing patent-infringement. We perform empirical evaluation showing superior performance of state-of-the-art models when using the novel loss function. The deployment of the system is underway at the Targeted Drug Delivery and Personalized Medicine labs. It will be utilized to generate targeted carriers of mRNA, providing a new method of drug delivery. The system is also producing non-patented candidates for industrial use, making it a valuable tool in the field of personalized medicine.

To improve the accuracy of clinical multivariate time series (MTS) classification (such as EEG and ECG) by a novel self-supervised paradigm that directly captures the dynamics between the different time series learned together to optimize the classification task. Labels in clinical datasets are very often insufficient. One way to address this challenge is leveraging self-supervision. This paradigm attempts to identify a supervisory signal inherent within a dataset to serve as a surrogate label. We present a novel form of self-supervision: dynamics of clinical MTS. Unlike other self-supervision methods, such as masking, that are intuitive but still heuristic, we suggest to learn a representation justified by Koopman theory. The latter was shown useful for representing clinical time series and can be used as a form of surrogate task to improve the clinical MTS classification. In the ECG task, we show that our proposed framework achieved higher sensitivity and specificity than the state-of-the-art (SOTA) baseline over numerous common diagnoses. For EEG abnormality classification, our proposed framework also achieved higher sensitivity and specificity than the SOTA baseline. All results are statistically significant. Our technique yields reliable clinical diagnosis in an empirical study employing signals from thousands of patients in multiple clinical tasks employing two types of clinical-grade sensors (ECG and EEG) as compared to the state-of-the-art machine learning. Leveraging time-series-dynamics self-supervision can help mitigate the lack of labels in clinical datasets used for training machine learning algorithms and significantly improve their performance. Specifically, the ECG system presented in this work is being trialed in hospitals, used by top cardiologists for patient diagnosis and treatment. We believe that the deployment of such cutting-edge technology will significantly improve the accuracy and speed of cardiac assessments.

Many texts, especially in chemistry and biology, describe complex processes. We focus on texts that describe a chemical reaction process and questions that ask about the process's outcome under different environmental conditions. To answer questions about such processes, one needs to understand the interactions between the different entities involved in the process and simulate their state transitions during the process execution under other conditions. We hypothesize that generating code and executing it to simulate the process will allow answering such questions. We, therefore, define a domain-specific language (DSL) to represent processes. We contribute to the community a unique dataset curated by chemists and annotated by computer scientists. The dataset is composed of process texts, simulation questions, and their corresponding computer codes represented by the DSL. We propose a neural program synthesis approach based on reinforcement learning with a novel state-transition semantic reward. The novel reward is based on the run-time semantic similarity between the predicted code and the reference code. This allows simulating complex process transitions and thus answering simulation questions. Our approach yields a significant boost in accuracy for simulation questions: we achieved 88% accuracy as opposed to 83% accuracy of the state-of-the-art neural program synthesis approaches and 54% accuracy of state-of-the-art end-to-end text-based approaches.

Recent global events have emphasized the importance of accelerating the drug discovery process. A way to deal with the issue is to use machine learning to increase the rate at which drugs are made available to the public. However, chemical labeled data for real-world applications is extremely scarce making traditional approaches less effective. A fruitful course of action for this challenge is to pretrain a model using related tasks with large enough datasets, with the next step being finetuning it for the desired task. This is challenging as creating these datasets requires labeled data or expert knowledge. To aid in solving this pressing issue, we introduce MISU - Molecular Inherent SUpervision, a unique method for pretraining graph neural networks for molecular property prediction. Our method leapfrogs past the need for labeled data or any expert knowledge by introducing three innovative components that utilize inherent properties of molecular graphs to induce information extraction at different scales, from the local neighborhood of an atom to substructures in the entire molecule. Our empirical results for six chemical-property-prediction tasks show that our method reaches state-of-the-art results compared to numerous baselines.

Consumer demand forecasting is of high importance for many e-commerce applications, including supply chain optimization, advertisement placement, and delivery speed optimization. However, reliable time series sales forecasting for e-commerce is difficult, especially during periods with many anomalies, as can often happen during pandemics, abnormal weather, or sports events. Although many time series algorithms have been applied to the task, prediction during anomalies still remains a challenge. In this work, we hypothesize that leveraging external knowledge found in world events can help overcome the challenge of prediction under anomalies. We mine a large repository of 40 years of world events and their textual representations. Further, we present a novel methodology based on transformers to construct an embedding of a day based on the relations of the day's events. Those embeddings are then used to forecast future consumer behavior. We empirically evaluate the methods over a large e-commerce products sales dataset, extracted from eBay, one of the world's largest online marketplaces. We show over numerous categories that our method outperforms state-of-the-art baselines during anomalies.

Our world is constantly evolving, and so is the content on the web. Consequently, our languages, often said to mirror the world, are dynamic in nature. However, most current contextual language models are static and cannot adapt to changes over time. In this work, we propose a temporal contextual language model called TempoBERT, which uses time as an additional context of texts. Our technique is based on modifying texts with temporal information and performing time masking - specific masking for the supplementary time information. We leverage our approach for the tasks of semantic change detection and sentence time prediction, experimenting on diverse datasets in terms of time, size, genre, and language. Our extensive evaluation shows that both tasks benefit from exploiting time masking.

Molecular lead optimization is an important task of drug discovery focusing on generating molecules similar to a drug candidate but with enhanced properties. Most prior works focused on optimizing a single property. However, in real settings, we wish to find molecules that satisfy multiple constraints, e.g., potency and safety. Simultaneously optimizing these constraints was shown to be difficult, mostly due to the lack of training examples satisfying all constraints. In this work, we present a novel approach for multi-property optimization. Unlike prior approaches, that require a large training set of pairs of a lead molecule and an enhanced molecule, our approach is unpaired. Our architecture learns a transformation for each property optimization separately, while constraining the latent embedding space between all transformations. This allows generating a molecule which optimizes multiple properties simultaneously. We present a novel adaptive loss which balances the separate transformations and stabilizes the optimization process. We evaluate our method on optimizing for two properties: dopamine receptor (DRD2) and drug likeness (QED), and show our method outperforms previous state-of-the-art, especially when training examples satisfying all constraints are sparse.

Molecular lead optimization is an important task of drug discovery focusing on generating novel molecules similar to a drug candidate but with enhanced properties. Prior works focused on supervised models requiring datasets of pairs of a molecule and an enhanced molecule. These approaches require large amounts of data and are limited by the bias of the specific examples of enhanced molecules. In this work, we present an unsupervised generative approach with a molecule-embedding component that maps a discrete representation of a molecule to a continuous space. The components are then coupled with a unique training architecture leveraging molecule fingerprints and applying double cycle constraints to enable both chemical resemblance to the original molecular lead while generating novel molecules with enhanced properties. We evaluate our method on multiple common molecular optimization tasks, including dopamine receptor (DRD2) and drug likeness (QED), and show our method outperforms previous state-of-the-art baselines. Moreover, we conduct thorough ablation experiments to show the effect and necessity of important components in our model. Furthermore, we demonstrate our method's ability to generate FDA-approved drugs it has never encountered before, such as Perazine and Clozapine, which are used to treat psychotic disorders, like Schizophrenia. The system is currently being deployed for use in the Targeted Drug Delivery and Personalized Medicine laboratories generating treatments using nanoparticle-based technology.

A significant number of event-related queries are issued in Web search. In this paper, we seek to improve retrieval performance by leveraging events and specifically target the classic task of query expansion. We propose a method to expand an event-related query by first detecting the events related to it. Then, we derive the candidates for expansion as terms semantically related to both the query and the events. To identify the candidates, we utilize a novel mechanism to simultaneously embed words and events in the same vector space. We show that our proposed method of leveraging events improves query expansion performance significantly compared with state-of-the-art methods on various newswire TREC datasets.

Temporal dynamic graphs are graphs whose topology evolves over time, with nodes and edges added and removed between different time snapshots. Embedding such graphs in a low-dimensional space is important for a variety of tasks, including graphs' similarities, time series trends analysis and anomaly detection, graph visualization, graph classification, and clustering. Despite the importance of the temporal element in these tasks, existing graph embedding methods focus on capturing the graph's nodes in a static mode and/or do not model the graph in its entirety in temporal dynamic mode. In this study, we present tdGraphEmbed, a novel temporal graph-level embedding approach that extend the random-walk based node embedding methods to globally embed both the nodes of the graph and its representation at each time step, thus creating representation of the entire graph at each step. Our approach was applied to graph similarity ranking, temporal anomaly detection, trend analysis, and graph visualizations tasks, where we leverage our temporal embedding in a fast and scalable way for each of the tasks. An evaluation of tdGraphEmbed on five real-world datasets shows that our approach can outperform state-of-the-art approaches used for graph embedding and node embedding in temporal graphs.

Personalization plays a key role in electronic commerce, adjusting the products presented to users through search and recommendations according to their personality and tastes. Current personalization efforts focus on the adaptation of product selections, while the description of a given product remains the same regardless of the user who views it. In this work, we propose an approach to personalize product descriptions according to the personality of an individual user. To the best of our knowledge, we are the first to address the problem of generating personalized product descriptions. We first learn to predict a user's personality based on past activity on an e-commerce website. Then, given a user personality, we propose an extractive summarization-based algorithm that selects the sentences to be used as part of a product description in accordance with the given personality. Our evaluation shows that user personality can be effectively learned from past e-commerce activity, while personalized descriptions can lead to a higher interest in the product and increased purchase likelihood.

Result ranking diversification has become an important issue for web search, summarization, and question answering. For more complex questions with multiple aspects, such as those in community-based question answering (CQA) sites, a retrieval system should provide a diversified set of relevant results, addressing the different aspects of the query, while minimizing redundancy or repetition. We present a new method, DRN , which learns novelty-related features from unlabeled data with minimal social signals, to emphasize diversity in ranking. Specifically, DRN parameterizes question-answer interactions via an LSTM representation, coupled with an extension of neural tensor network, which in turn is combined with a novelty-driven sampling approach to automatically generate training data. DRN provides a novel and general approach to complex question answering diversification and suggests promising directions for search improvements.

Product descriptions play an important role in the e-commerce ecosystem, conveying to buyers information about a merchandise they may purchase. Yet, on leading e-commerce websites, with high volumes of new items offered for sale every day, product descriptions are often lacking or missing altogether. Moreover, many descriptions include information that holds little value and sometimes even disrupts buyers, in an attempt to draw attention and purchases. In this work, we suggest to mitigate these issues by generating short crowd-based product descriptions from user reviews . We apply an extractive approach, where review sentences are used in their original form to compose the product description. At the core of our method is a supervised approach to identify candidate review sentences suitable to be used as part of a description. Our analysis, based on data from both the Fashion and Motors domains, reveals the top reasons for review sentences being unsuitable for the product's description and these are used, in turn, as part of a deep multi-task learning architecture. We then diversify the set of candidates by removing redundancies and, at the final step, select the top candidates to be included in the description. We compare different methods for each step and also conduct an end-to-end evaluation, based on rating from professional annotators, showing the generated descriptions are of high quality.

Designing a new drug is a lengthy and expensive process. As the space of potential molecules is very large (10 23 - 10 60 ), a common technique during drug discovery is to start from a molecule which already has some of the desired properties. An interdisciplinary team of scientists generates hypothesis about the required changes to the prototype. In this work, we develop an algorithmic unsupervised-approach that automatically generates potential drug molecules given a prototype drug. We show that the molecules generated by the system are valid molecules and significantly different from the prototype drug. Out of the compounds generated by the system, we identified 35 FDA-approved drugs. As an example, our system generated Isoniazid - one of the main drugs for Tuberculosis. The system is currently being deployed for use in collaboration with pharmaceutical companies to further analyze the additional generated molecules.

Electronic commerce continues to gain popularity in recent years. On eBay, one of the largest on-line marketplaces in the world, millions of new listings (items) are submitted by a variety of sellers every day. This renders a rich diverse inventory characterized by a particularly long tail. In addition, many items in the inventory lack basic structured information, such as product identifiers, brand, category, and other properties, due to sellers' tendency to input unstructured information only, namely title and description. Such inventory therefore requires a handful of large-scale solutions to assist in organizing the data and gaining business insights. In 2016, eBay acquired SalesPredict to help structure its unstructured data. In this proposed presentation, we will share the story of a research startup from its inception until its acquisition and integration as eBay's data science team. We will review the numerous challenges from research and engineering perspectives of a startup and the principal challenges the eBay data science organization deals with today. These include the identification of duplicate, similar, and related products; the extraction of name-value attributes from item titles and descriptions; the matching of items entered by sellers to catalog products; the ranking of item titles based on their likelihood to serve as "good" product titles; and the creation of "browse node" pages to address complex search queries from potential buyers. We will describe how the eBay data science team approaches these challenges and some of the solutions already launched to production. These solutions involve the use of large-scale machine learning, information retrieval, and natural language processing techniques, and should therefore be of interest to the SIGIR audience at large.

Web content increasingly reflects the current state of the physical and social world, manifested both in traditional news media sources along with user-generated publishing sites such as Twitter, Foursquare, and Facebook. At the same time, web searching increasingly reflects problems grounded in the real world. As a result of this blending of the web with the real world, we observe that the web, both in its composition and use, has incorporated many of the dynamics of the real world. Few of the problems associated with searching dynamic collections are well understood, such as defining time-sensitive relevance, understanding user query behavior over time and understanding why certain web content changes.

Accurate prediction of changing web page content improves a variety of retrieval and web related components. For example, given such a prediction algorithm one can both design a better crawling strategy that only recrawls pages when necessary as well as a proactive mechanism for personalization that pushes content associated with user revisitation directly to the user. While many techniques for modeling change have focused simply on past change frequency, our work goes beyond that by additionally studying the usefulness in page change prediction of: the page's content; the degree and relationship among the prediction page's observed changes; the relatedness to other pages and the similarity in the types of changes they undergo. We present an expert prediction framework that incorporates the information from these other signals more effectively than standard ensemble or basic relational learning techniques. In an empirical analysis, we find that using page content as well as related pages significantly improves prediction accuracy and compare it to common approaches. We present numerous similarity metrics to identify related pages and focus specifically on measures of temporal content similarity. We observe that the different metrics yield related pages that are qualitatively different in nature and have different effects on the prediction performance.

The World Wide Web is highly dynamic and is constantly evolving to cover the latest information about the physical and social updates in the world. At the same time, the changes in web contents are entangled with new information needs and time-sensitive user interactions with information sources. To address these temporal information needs effectively, it is essential for the search engines to model web dynamics and understand the changes in user behavior over time that are caused by them.

We describe and evaluate methods for learning to forecast forthcoming events of interest from a corpus containing 22 years of news stories. We consider the examples of identifying significant increases in the likelihood of disease outbreaks, deaths, and riots in advance of the occurrence of these events in the world. We provide details of methods and studies, including the automated extraction and generalization of sequences of events from news corpora and multiple web resources. We evaluate the predictive power of the approach on real-world events withheld from the system.

The queries people issue to a search engine and the results clicked following a query change over time. For example, after the earthquake in Japan in March 2011, the query japan spiked in popularity and people issuing the query were more likely to click government-related results than they would prior to the earthquake. We explore the modeling and prediction of such temporal patterns in Web search behavior. We develop a temporal modeling framework adapted from physics and signal processing and harness it to predict temporal patterns in search behavior using smoothing, trends, periodicities, and surprises. Using current and past behavioral data, we develop a learning procedure that can be used to construct models of users' Web search activities. We also develop a novel methodology that learns to select the best prediction model from a family of predictive models for a given query or a class of queries. Experimental results indicate that the predictive models significantly outperform baseline models that weight historical evidence the same for all queries. We present two applications where new methods introduced for the temporal modeling of user behavior significantly improve upon the state of the art. Finally, we discuss opportunities for using models of temporal dynamics to enhance other areas of Web search and information retrieval.

Query auto-completion (QAC) is a common feature in modern search engines. High quality QAC candidates enhance search experience by saving users time that otherwise would be spent on typing each character or word sequentially.

The problem we tackle in this work is, given a present news event, to generate a plausible future event that can be caused by the given event. We present a new methodology for modeling and predicting such future news events using machine learning and data mining techniques. Our Pundit algorithm generalizes examples of causality pairs to infer a causality predictor. To obtain precise labeled causality examples, we mine 150 years of news articles, and apply semantic natural language modeling techniques to titles containing certain predefined causality patterns. For generalization, the model uses a vast amount of world knowledge ontologies mined from LinkedData, containing ~200 datasets with approximately 20 billion relations. Empirical evaluation on real news articles shows that our Pundit algorithm reaches a human-level performance.

User behavior on the Web changes over time. For example, the queries that people issue to search engines, and the underlying informational goals behind the queries vary over time. In this paper, we examine how to model and predict this temporal user behavior. We develop a temporal modeling framework adapted from physics and signal processing that can be used to predict time-varying user behavior using smoothing and trends. We also explore other dynamics of Web behaviors, such as the detection of periodicities and surprises. We develop a learning procedure that can be used to construct models of users' activities based on features of current and historical behaviors. The results of experiments indicate that by using our framework to predict user behavior, we can achieve significant improvements in prediction compared to baseline models that weight historical evidence the same for all queries. We also develop a novel learning algorithm that explicitly learns when to apply a given prediction model among a set of such models. Our improved temporal modeling of user behavior can be used to enhance query suggestions, crawling policies, and result ranking.

Obtaining judgments from human raters is a vital part in the design of search engines' evaluation. Today, a discrepancy exists between judgment acquisition from raters (training phase) and use of the responses for retrieval evaluation (evaluation phase). This discrepancy is due to the inconsistency between the representation of the information in both phases. During training, raters are requested to provide a relevance score for an individual result in the context of a query, whereas the evaluation is performed on ordered lists of search results, with the results' relative position (compared to other results) taken into account. As an alternative to the practice of learning to rank using relevance judgments for individual search results, more and more focus has recently been diverted to the theory and practice of learning from answers to combinatorial questions about sets of search results. That is, users, during training, are asked to rank small sets (typically pairs).

Computing the degree of semantic relatedness of words is a key functionality of many language applications such as search, clustering, and disambiguation. Previous approaches to computing semantic relatedness mostly used static language resources, while essentially ignoring their temporal aspects. We believe that a considerable amount of relatedness information can also be found in studying patterns of word usage over time. Consider, for instance, a newspaper archive spanning many years. Two words such as "war" and "peace" might rarely co-occur in the same articles, yet their patterns of use over time might be similar. In this paper, we propose a new semantic relatedness model, Temporal Semantic Analysis (TSA), which captures this temporal information. The previous state of the art method, Explicit Semantic Analysis (ESA), represented word semantics as a vector of concepts. TSA uses a more refined representation, where each concept is no longer scalar, but is instead represented as time series over a corpus of temporally-ordered documents. To the best of our knowledge, this is the first attempt to incorporate temporal evidence into models of semantic relatedness. Empirical evaluation shows that TSA provides consistent improvements over the state of the art ESA results on multiple benchmarks.

The novel task we aim at in this work is to predict top terms that will prominently appear in the future news. This is a difficult task that nobody attempted before, as far as we know. We present a novel methodology for using patterns of user queries to predict future events. Query history is obtained from Web resources such as Google trends. In order to predict whether a term will appear in tomorrow's news, we examine if the terms in today's queries indicated this term in the past. We provide empirical support for the effectiveness of our method by showing its prediction power on news archives.