ABSTRACT Chronic kidney disease (CKD) is a serious global health threat. At the terminal stage, kidney function is nearly completely lost. Therefore, predicting the development of CKD based on a patient's visits can enable doctors to intervene early and delay the disease's progression. In this paper, we propose a three‐stage prediction model named Imputation‐Capture‐Prediction (ICP) and based on the Transformer architecture, for chronic kidney disease (CKD) using electronic health records (EHRs). The first stage is to address the missing data problem in EHR, and ICP employs a two‐stage imputation method, using the deep learning method SAITS module after recent padding. The second stage is designed to better capture this temporal dependency and the relationships between features, where ICP incorporates a two‐branch architecture and introduces two modules: Time‐Aware Convolution (TC) and Dynamic‐Static‐Medical Graph Attention Network (DSMGAT), to extract diverse feature information. The TC module is designed to capture the relationships within visit records, accounting for the unequal lengths of visit intervals while emphasizing the importance of recent records. The DSMGAT module, on the other hand, considers various categories of record features, using a Graph Attention Network (GAT) with learnable weights to model the relationships among them. Then we use a Feed‐Forward Network to predict the estimated glomerular filtration rate (eGFR). To evaluate the effectiveness of our method, we compared it with several advanced approaches using a real EHR dataset, TFHCKD. The Mean Absolute Error (MAE) and Mean Squared Error (MSE) were 0.0344 and 0.0028, respectively, demonstrating a significant improvement over existing methods.

Optimization is ubiquitous. While derivative-based algorithms have been powerful tools for various problems, the absence of gradient imposes challenges on many real-world applications. In this work, we propose Optimization by PROmpting (OPRO), a simple and effective approach to leverage large language models (LLMs) as optimizers, where the optimization task is described in natural language. In each optimization step, the LLM generates new solutions from the prompt that contains previously generated solutions with their values, then the new solutions are evaluated and added to the prompt for the next optimization step. We first showcase OPRO on linear regression and traveling salesman problems, then move on to our main application in prompt optimization, where the goal is to find instructions that maximize the task accuracy. With a variety of LLMs, we demonstrate that the best prompts optimized by OPRO outperform human-designed prompts by up to 8% on GSM8K, and by up to 50% on Big-Bench Hard tasks. Code at https://github.com/google-deepmind/opro.

Large language models (LLMs) often generate content that contains factual errors when responding to fact-seeking prompts on open-ended topics. To benchmark a model’s long-form factuality in open domains, we first use GPT-4 to generate LongFact, a prompt set comprising thousands of questions spanning 38 topics. We then propose that LLM agents can be used as automated evaluators for long-form factuality through a method which we call Search-Augmented Factuality Evaluator (SAFE). SAFE utilizes an LLM to break down a long-form response into a set of individual facts and to evaluate the accuracy of each fact using a multi-step reasoning process comprising sending search queries to Google Search and determining whether a fact is supported by the search results. Furthermore, we propose extending F1 score as an aggregated metric for long-form factuality. To do so, we balance the percentage of supported facts in a response (precision) with the percentage of provided facts relative to a hyperparameter representing a user’s preferred response length (recall). Empirically, we demonstrate that LLM agents can outperform crowdsourced human annotators—on a set of∼16k individual facts, SAFE agrees with crowdsourced human annotators 72% of the time, and on a random subset of 100 disagreement cases, SAFE wins 76% of the time. At the same time, SAFE is more than 20 times cheaper than human annotators. We also benchmark thirteen language models on LongFact across four model families (Gemini, GPT, Claude, and PaLM-2), finding that larger language models generally achieve better long-form factuality. LongFact, SAFE, and all experimental code are available at https://github.com/google-deepmind/long-form-factuality.

Recent advances in machine learning have leveraged dramatic increases in computational power, a trend expected to continue in the future. This paper introduces the first Hyperscale Hardware Optimized Neural Architecture Search (H2O-NAS) to automatically design accurate and performant machine learning models tailored to the underlying hardware architecture. H2O-NAS consists of three key components: a new massively parallel “one-shot” search algorithm with intelligent weight sharing, which can scale to search spaces of O(10280) and handle large volumes of production traffic; hardware-optimized search spaces for diverse ML models on heterogeneous hardware; and a novel two-phase hybrid performance model and a multi-objective reward function optimized for large scale deployments.

Jerry Wei, Le Hou, Andrew Lampinen, Xiangning Chen, Da Huang, Yi Tay, Xinyun Chen, Yifeng Lu, Denny Zhou, Tengyu Ma, Quoc Le. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023.

Transformers are central to recent successes in natural language processing and computer vision. Transformers have a mostly uniform backbone where layers alternate between feed-forward and self-attention in order to build a deep network. Here we investigate this design choice and find that more complex blocks that have different permutations of layer primitives can be more efficient. Using this insight, we develop a complex block, named Brainformer, that consists of a diverse sets of layers such as sparsely gated feed-forward layers, dense feed-forward layers, attention layers, and various forms of layer normalization and activation functions. Brainformer consistently outperforms the state-of-the-art dense and sparse Transformers, in terms of both quality and efficiency. A Brainformer model with 8 billion activated parameters per token demonstrates 2x faster training convergence and 5x faster step time compared to its GLaM counterpart. In downstream task evaluation, Brainformer also demonstrates a 3% higher SuperGLUE score with fine-tuning compared to GLaM with a similar number of activated parameters. Finally, Brainformer largely outperforms a Primer dense model derived with NAS with similar computation per token on fewshot evaluations.

We present a method to formulate algorithm discovery as program search, and apply it to discover optimization algorithms for deep neural network training. We leverage efficient search techniques to explore an infinite and sparse program space. To bridge the large generalization gap between proxy and target tasks, we also introduce program selection and simplification strategies. Our method discovers a simple and effective optimization algorithm, $\textbf{Lion}$ ($\textit{Evo$\textbf{L}$ved S$\textbf{i}$gn M$\textbf{o}$me$\textbf{n}$tum}$). It is more memory-efficient than Adam as it only keeps track of the momentum. Different from adaptive optimizers, its update has the same magnitude for each parameter calculated through the sign operation. We compare Lion with widely used optimizers, such as Adam and Adafactor, for training a variety of models on different tasks. On image classification, Lion boosts the accuracy of ViT by up to 2\% on ImageNet and saves up to 5x the pre-training compute on JFT. On vision-language contrastive learning, we achieve 88.3\% $\textit{zero-shot}$ and 91.1\% $\textit{fine-tuning}$ accuracy on ImageNet, surpassing the previous best results by 2\% and 0.1\%, respectively. On diffusion models, Lion outperforms Adam by achieving a better FID score and reducing the training compute by up to 2.3x. For autoregressive, masked language modeling, and fine-tuning, Lion exhibits a similar or better performance compared to Adam. Our analysis of Lion reveals that its performance gain grows with the training batch size. It also requires a smaller learning rate than Adam due to the larger norm of the update produced by the sign function. Additionally, we examine the limitations of Lion and identify scenarios where its improvements are small or not statistically significant.

The mixture proportions of pretraining data domains (e.g., Wikipedia, books, web text) greatly affect language model (LM) performance. In this paper, we propose Domain Reweighting with Minimax Optimization (DoReMi), which first trains a small proxy model using group distributionally robust optimization (Group DRO) over domains to produce domain weights (mixture proportions) without knowledge of downstream tasks. We then resample a dataset with these domain weights and train a larger, full-sized model. In our experiments, we use DoReMi on a 280M-parameter proxy model to set the domain weights for training an 8B-parameter model (30x larger) more efficiently. On The Pile, DoReMi improves perplexity across all domains, even when it downweights a domain. DoReMi improves average few-shot downstream accuracy by 6.5% points over a baseline model trained using The Pile's default domain weights and reaches the baseline accuracy with 2.6x fewer training steps. On the GLaM dataset, DoReMi, which has no knowledge of downstream tasks, even matches the performance of using domain weights tuned on downstream tasks.

Lidars and cameras are critical sensors that provide complementary information for 3D detection in autonomous driving. While prevalent multi-modal methods [34], [36] simply decorate raw lidar point clouds with camera features and feed them directly to existing 3D detection models, our study shows that fusing camera features with deep lidar features instead of raw points, can lead to better performance. However, as those features are often augmented and aggregated, a key challenge in fusion is how to effectively align the transformed features from two modalities. In this paper, we propose two novel techniques: InverseAug that inverses geometric-related augmentations, e.g., rotation, to enable accurate geometric alignment between lidar points and image pixels, and LearnableAlign that leverages cross-attention to dynamically capture the correlations between image and lidar features during fusion. Based on InverseAug and LearnableAlign, we develop a family of generic multi-modal 3D detection models named DeepFusion, which is more accurate than previous methods. For example, DeepFusion improves Point-Pillars, CenterPoint, and 3D-MAN baselines on Pedestrian detection for 6.7,8.9, and 6.2 LEVEL_2 APH, respectively. Notably, our models achieve state-of-the-art performance on Waymo Open Dataset, and show strong model robustness against input corruptions and out-of-distribution data. Code will be publicly available at https://github.com/tensorflow/lingvo.

The best neural architecture for a given machine learning problem depends on many factors: not only the complexity and structure of the dataset, but also on resource constraints including latency, compute, energy consumption, etc. Neural architecture search (NAS) for tabular datasets is an important but under-explored problem. Previous NAS algorithms designed for image search spaces incorporate resource constraints directly into the reinforcement learning (RL) rewards. However, for NAS on tabular datasets, this protocol often discovers suboptimal architectures. This paper develops TabNAS, a new and more effective approach to handle resource constraints in tabular NAS using an RL controller motivated by the idea of rejection sampling. TabNAS immediately discards any architecture that violates the resource constraints without training or learning from that architecture. TabNAS uses a Monte-Carlo-based correction to the RL policy gradient update to account for this extra filtering step. Results on several tabular datasets demonstrate the superiority of TabNAS over previous reward-shaping methods: it finds better models that obey the constraints.

Nowadays, it is a rapidly rising demand yet challenging issue to run large-scale applications on shared infrastructures such as data centers and clouds with low execution latency and high resource utilization. This paper reports our experience with Swift, a system capable of efficiently running real-time and interactive data processing jobs at cloud scale. Taking directed acyclic graph DAG as the job model, Swift achieves the design goal by three new mechanisms: 1 fine-grained scheduling that can efficiently partition a job into graphlets i.e., sub-graphs based on new shuffle heuristics and that does scheduling in the unit of graphlet, thus avoiding resource fragmentation and waste, 2 adaptive memory-based in-network shuffling that reduces IO overhead and data transfer time by doing shuffle in memory and allowing jobs to select the most efficient way to fulfill shuffling, and 3 lightweight fault tolerance and recovery that only prolong the whole job execution time slightly with the help of timely failure detection and fine-grained failure recovery. Experimental results show that Swift can achieve an average speedup of 2.11× on TPC-H, and 14.18× on Terasort when compared with Spark. Swift has been deployed in production, supporting as many as 140,000 executors and processing millions of jobs per day. Experiments with production traces show that Swift outperforms JetScope and Bubble Execution by 2.44× and 1.23× respectively.

Pervasive needs for data explorations at all scales have populated modern distributed platforms with workloads of different characteristics. The growing complexities and diversities have thereafter imposed distinct challenges to execute them on shared clusters in corporate or public clouds. This paper presents Fangorn, an adaptive execution framework built on an enriched graph model. As the underlying infrastructure for core computation platforms at Alibaba, Fangorn supports various execution modes and caters to heterogeneous workloads. With the capability to orchestrate graph executions with both long-running and requested-on-demand resources at the same time, Fangorn allows exploration of tradeoffs between latency and resource efficiency, for jobs of all scales. By modeling distributed job executions as mutable graphs with pluggable components, Fangorn offers a systematic framework to adjust job executions adaptively, according to data statistics collected during run-time. Fangorn supports an array of different computation engines ranging from relational to deep learning, and is fully deployed on production clusters across Alibaba. It manages tens of millions of distributed jobs daily, with job size scaling from one to half-million.

Outlier detection is one of the most important tasks to keep your processes in control. Unawareness of critical anomalies can lead to exhausting expenses, hence, it is highly beneficial to treat process failures as soon as possible. However, anomalies are difficult to detect due to their rarity though they occur too often to neglect the necessity of its detection. Even if the detection problem is solved, the treatment of singular anomalies and the adjustment of the process based on each abnormal trace is tedious and costly regarding time and money. To increase the efficiency of later anomaly treatment, we propose a novel strategy to detect collective anomalies. However, this is not equivalent to anomaly clustering as a post-processing step. TOAD orders process instances by similarity and detects abnormal accumulations of deviating cases. These collections are abnormal due to their aggregated behavior. Assuming that similar deviations are caused by the same reason, the treatment of such an anomaly is more cost-efficient than the handling of deviating singletons. Applying TOAD to an event log yields a ranking of significant, temporally abnormal trace collections, that provide a baseline for further analysis.

Density-based clustering algorithms are commonly adopted when arbitrarily shaped clusters exist. Usually, they do not need to know the number of clusters in prior, which is a big advantage. Conventional density-based approaches such as DBSCAN, utilize two parameters to define density. Recently, novel density-based clustering algorithms are proposed to reduce the problem complexity to the use of a single parameter k by utilizing the concepts of k Nearest Neighbor (kNN) and Reverse k Nearest Neighbor (RkNN) to define density. However, those kNN-based approaches are either ineffective or inefficient. In this paper, we present a new clustering algorithm KNNAC, which only requires computing the densities for a chosen subset of points due to the use of active core detection. We empirically show that, compared to other nearest neighbor based clustering approaches (e.g., RECORD, IS-DBSCAN, etc.), KNNAC can provide competitive performance while taking a fraction of the runtime.

Existing clustering algorithms aim at identifying clusters from a single dataset. However, many applications generate a series of datasets. For example, scientists need to repeat an experiment many times to ensure reproducibility; sensors collect information day after day. In such scenarios, we need to identify clusters separately from a large number of datasets, which can contain an unknown number of clusters with various densities and shapes.Density-based clustering algorithms are commonly used in identifying arbitrary shaped clusters when the cluster number is unknown. Most density-based clustering algorithms are "DBSCAN-alike", where clusters are formed by connecting consecutive high dense regions. Therefore, points are grouped as one cluster as long as they are densely connected. When the distribution shape of points is changed across different datasets, parameter tuning on each dataset is necessary to obtain proper results, which is time-consuming.In this work, we developed a new kNN density-based clustering algorithm, which does not adopt the DBSCAN paradigm. Instead, we identify clusters by maximizing the intra-cluster similarities, which are estimated using: 1) the probability that two points belong to the same cluster; 2) the probability that a point is a cluster center. The kNN concept and minimum spanning tree are used to compute both probabilities. Our approach is capable of extracting clusters in arbitrary shapes using the single parameter k, and can handle a series of datasets with less parameter tuning effort. Experiments on both synthetic and real-world datasets show that our approach outperforms other recent kNN clustering algorithms.

Neural networks are sensitive to hyper-parameter and architecture choices. Automated Machine Learning (AutoML) is a promising paradigm for automating these choices. Current ML software libraries, however, are quite limited in handling the dynamic interactions among the components of AutoML. For example, efficient NAS algorithms, such as ENAS and DARTS, typically require an implementation coupling between the search space and search algorithm, the two key components in AutoML. Furthermore, implementing a complex search flow, such as searching architectures within a loop of searching hardware configurations, is difficult. To summarize, changing the search space, search algorithm, or search flow in current ML libraries usually requires a significant change in the program logic. In this paper, we introduce a new way of programming AutoML based on symbolic programming. Under this paradigm, ML programs are mutable, thus can be manipulated easily by another program. As a result, AutoML can be reformulated as an automated process of symbolic manipulation. With this formulation, we decouple the triangle of the search algorithm, the search space and the child program. This decoupling makes it easy to change the search space and search algorithm (without and with weight sharing), as well as to add search capabilities to existing code and implement complex search flows. We then introduce PyGlove, a new Python library that implements this paradigm. Through case studies on ImageNet and NAS-Bench-101, we show that with PyGlove users can easily convert a static program into a search space, quickly iterate on the search spaces and search algorithms, and craft complex search flows to achieve better results.

Abstract OPTICS is a popular tool to analyze the clustering structure of a dataset visually. The created two-dimensional plots indicate very dense areas and cluster candidates in the data as troughs. Each horizontal slice represents an outcome of a density-based clustering specified by the height as the density threshold for clusters. However, in very dynamic and rapidly changing applications, a complex and finely detailed visualization slows down the knowledge discovery. Instead, a framework that provides fast but coarse insights is required to point out structures in the data quickly. The user can then control the direction he wants to put emphasize on for refinement. We develop AMTICS as a novel and efficient divide-and-conquer approach to pre-cluster data in distributed instances and align the results in a hierarchy afterward. An interactive online phase ensures a low complexity while giving the user full control over the partial cluster instances. The offline phase reveals the current data clustering structure with low complexity and at any time.

In this work, we investigate the opposite question of frequent itemset mining: what patterns occurred less than a given minimum support in a transactional database? This question, known as infrequent itemset mining, is important in fields such as medical science, security, finance and scientific research. Frequent patterns represent expected or obvious information while infrequent patterns are those unexpected behaviors and are more interesting in some applications. For example, health-care needs to identify sporadic but lethal crossover effects. Security agents have to uncover infrequent associative fraud indicators. Existing infrequent itemset mining approaches are time-consuming. Furthermore, extracting all infrequent patterns might suffer from the redundant problem. In this paper, we study the two factors that affect the performance of itemset mining tasks. The concept of closed itemset is applied for infrequent patterns to reduce the number of returned patterns. An efficient closed infrequent itemset mining approach is proposed which combines both bottom-up and top-down traversing strategies. Extensive experimental results show that a simple algorithm based on our framework, without using advanced data structure or pruning techniques, can still be significantly more efficient when compared with other approaches.

Bioresorbable Vascular Scaffolds (BVS) are currently one of the most frequently-used type of stent during percutaneous coronary intervention. It's very important to conduct struts malappostion analysis during operation. Currently, BVS malappostion analysis in intravascular optical coherence tomography (IVOCT) images is mainly conducted manually, which is labor intensive and time consuming. In our previous work, a novel framework was presented to automatically detect and segment BVS struts for malappostion analysis. However, limited by the detection performance, the framework faced some challenges under complex background. In this paper, we proposed a robust BVS struts detection method based on Region-based Fully Convolutional Network (R-FCN). The detection model mainly consisted of two modules: 1) a Region Proposal Network (RPN), used to extract struts region of interest (ROIs) in the image and, 2) a detection module, used to classify the ROIs and regress a bounding box for each ROI. The network was initialized by pre-trained ImageNet model and then trained based on our labeled data which contained 1231 IVOCT images. Tested on a total of 480 IVOCT images with 4096 BVS struts, our method achieved 97.9% true positive rate with 4.79% false positive rate. It concludes that the proposed method is efficient and robust for BVS struts detection.

We reduce the computational cost of Neural AutoML with transfer learning. AutoML relieves human effort by automating the design of ML algorithms. Neural AutoML has become popular for the design of deep learning architectures, however, this method has a high computation cost. To address this we propose Transfer Neural AutoML that uses knowledge from prior tasks to speed up network design. We extend RL-based architecture search methods to support parallel training on multiple tasks and then transfer the search strategy to new tasks. On language and image classification data, Transfer Neural AutoML reduces convergence time over single-task training by over an order of magnitude on many tasks.

Bioresorbable Vascular Scaffolds (BVS) are the most promising type of stent in percutaneous coronary intervention. For accurate BVS struts apposition assessment, intravascular optical coherence tomography (IVOCT) is the state-of-the-art imaging modality. However, manual analysis for IVOCT frames is time consuming and labor intensive. In this paper, we propose an automatic method for BVS struts center and region detection based on Adaboost algorithm and Haar-like features. Then, dynamic programming algorithm is applied to segment the contour of BVS struts. Based on the segmentation results, the apposed or malapposed struts can be automatically distinguished. By comparing the manual and automatic detection and segmentation results, our method correctly detected and segmented 87.7% of 4029 BVS struts with 18.6% false positives. The average Dice's coefficient for the correctly detected struts was 0.78. In conclusion, the evaluation suggested that this method is accurate and robust for BVS struts detection and segmentation.

This paper address the problem of temporal pattern mining from multiple data streams containing temporal events. Temporal events are considered as real world events aligned with comprehensive starting and ending timing information rather than simple integer timestamps. Predefined relations, such as "before" and "after", describe the heterogeneous relationships hidden in temporal data with limited diversity. In this work, the relationships among events are learned dynamically from the temporal information. Each event is treated as an object with a label and numerical attributes. An online-offline model is used as the primary structure for analyzing the evolving multiple streams. Different distance functions on temporal events and sequences can be applied depending on the application scenario. A prefix tree is introduced for a fast incremental pattern update.

Almost all activities observed in nowadays applications are correlated with a timing sequence. Users are mainly looking for interesting sequences out of such data. Sequential pattern mining algorithms aim at nding frequent sequences. Usually, the mined activities have timing durations that represent time intervals between their starting and ending points. Most sequential pattern mining approaches dealt with such activities as a single point event and thus lost many valuable information in the collected patterns. We present the PIVOTMiner, an ecient interval-based sequential pattern mining algorithm using a geometric representation of intervals. The interestingness level is not necessarily positively correlated with the frequency of the patterns. In many applications, users are seeking for rare patterns that considerably deviate from the majority. Simply delivering the bottom-k patterns does not guarantee their high outlierness (or deviation) from the frequent ones. We propose additionally the PIVOTRanker, the rst scalable algorithm for ranking rare interval-based sequential patterns based on their outlierness. Our experimental results on both synthetic and real-world datasets show that PIVOTMiner spends considerably less time than two state-of-the-art competitors, and that PIVOTRanker delivers a meaningful and useful ranking of rare patterns.

Almost all activities observed in nowadays applications are correlated with a timing sequence. Users are mainly looking for interesting sequences out of such data. Sequential pattern mining algorithms aim at finding frequent sequences. Usually, the mined activities have timing durations that represent time intervals between their starting and ending points. The majority of sequential pattern mining approaches dealt with such activities as a single point event and thus lost valuable information in the collected patterns. Recently, some approaches have carefully considered this interval-based nature of the events, but they have major limitations. They concentrate only on the order of events without taking the durations of the gaps between them into account and usually employ a binary representation to describe patterns. To resolve these problems, we propose the PIVOTMiner, an interval-based data mining algorithm using a geometric representation approach of intervals. Noisy events can be served with the geometric representation and a fuzzy set can be retrieved from the geometric patterns. PIVOTMiner can flexibly work on data presented as any number of not necessarily aligned interval sequences and in particular can utilize data presented as single interval sequence stream without the need to create samples. Our experimental results on both synthetic and real-world smart home datasets show that the information presented in our mined patterns are richer than those of most state-of-the-art algorithms while spending considerably smaller running times.

In this paper, we present an improved method for detecting LSB matching steganography in gray-scale image. Our improvements focus on three aspects: (1) instead of using the amplitude of local extrema of the image's histogram in the previous work, we turn to considering the sum of the amplitude of each point in the histogram; (2) incorporating the calibration (downsample) technique with the current method; (3) the sum/difference image (which is defined as the sum or difference of two adjacent pixels in the original image) is taken into consideration to provide additional statistical features. Extensive experimental results show that the novel steganalyzer out-performs the previous ones.