Large language models (LLMs) contain substantial factual knowledge which is commonly elicited by multiple-choice question-answering prompts. Internally, such models process the prompt through multiple transformer layers, building varying representations of the problem within its hidden states. Ultimately, however, only the hidden state corresponding to the final layer and token position is used to predict the answer label. In this work, we propose instead to learn a small separate neural network predictor module on a collection of training questions, that take the hidden states from all the layers at the last temporal position as input and outputs predictions. In effect, such a framework disentangles the representational abilities of LLMs from their predictive abilities. On a collection of hard benchmarks, our method achieves considerable improvements in performance, sometimes comparable to supervised fine-tuning procedures, but at a fraction of the computational cost.

Online deep learning tackles the challenge of learning from data streams by balancing two competing goals: fast learning and deep learning. However, existing research primarily emphasizes deep learning solutions, which are more adept at handling the ”deep” aspect than the ”fast” aspect of online learning. In this work, we introduce an alternative paradigm through a hybrid multilearner approach. We begin by developing a fast online logistic regression learner, which operates without relying on backpropagation. It leverages closed-form recursive updates of model parameters, efficiently addressing the fast learning component of the online learning challenge. This approach is further integrated with a cascaded multilearner design, where shallow and deep learners are co-trained in a cooperative, synergistic manner to solve the online learning problem. We demonstrate that this approach achieves state-of-the-art performance on standard online learning datasets. We make our code available: \url{https://github.com/AntonValk/MODL}

Recent advances in recommender systems have shown that relying on graph filters, such as the normalized item-item adjacency matrix and the ideal low-pass filter yields competitive performance and scales better than Graph Convolutional Networks-based solutions. However, these solutions require centralizing user data, which raises concerns over data privacy, security, and the monopolization of user data by a few actors. To address those concerns, we propose PriviRec and PriviRec-k, two complementary recommendation frameworks. In PriviRec, we show that it is possible to decompose widely used filters so that they can be computed in a distributed setting using Secure Aggregation and a distributed version of the Randomized Power Method, without revealing individual users contributions. PriviRec-k extends this approach by having users securely aggregate low-rank projections of their contributions, enabling a tunable balance between communication overhead and recommendation accuracy. We demonstrate theoretically as well as experimentally on Gowalla, Yelp2018, and Amazon-Book that our methods achieve performance comparable to centralized state-of-the-art recommender systems and superior to decentralized ones, while preserving confidentiality and low communication and computational overheads.

A significant challenge in maintaining real-world machine learning models is responding to the continuous and unpredictable evolution of data. Most practitioners are faced with the difficult question: when should I retrain or update my machine learning model? This seemingly straightforward problem is particularly challenging for three reasons: 1) decisions must be made based on very limited information - we usually have access to only a few examples, 2) the nature, extent, and impact of the distribution shift are unknown, and 3) it involves specifying a cost ratio between retraining and poor performance, which can be hard to characterize. Existing works address certain aspects of this problem, but none offer a comprehensive solution. Distribution shift detection falls short as it cannot account for the cost trade-off; the scarcity of the data, paired with its unusual structure, makes it a poor fit for existing offline reinforcement learning methods, and the online learning formulation overlooks key practical considerations. To address this, we present a principled formulation of the retraining problem and propose an uncertainty-based method that makes decisions by continually forecasting the evolution of model performance evaluated with a bounded metric. Our experiments, addressing classification tasks, show that the method consistently outperforms existing baselines on 7 datasets. We thoroughly assess its robustness to varying cost trade-off values and mis-specified cost trade-offs.

Koopman operator theory provides a framework for nonlinear dynamical system analysis and time-series forecasting by mapping dynamics to a space of real-valued measurement functions, enabling a linear operator representation. Despite the advantage of linearity, the operator is generally infinite-dimensional. Therefore, the objective is to learn measurement functions that yield a tractable finite-dimensional Koopman operator approximation. In this work, we establish a connection between Koopman operator approximation and linear Recurrent Neural Networks (RNNs), which have recently demonstrated remarkable success in sequence modeling. We show that by considering an extended state consisting of lagged observations, we can establish an equivalence between a structured Koopman operator and linear RNN updates. Building on this connection, we present SKOLR, which integrates a learnable spectral decomposition of the input signal with a multilayer perceptron (MLP) as the measurement functions and implements a structured Koopman operator via a highly parallel linear RNN stack. Numerical experiments on various forecasting benchmarks and dynamical systems show that this streamlined, Koopman-theory-based design delivers exceptional performance. Our code is available at: https://github.com/networkslab/SKOLR.

Recent years have witnessed the emergence of a spectrum of foundation models, covering a broad range of capabilities and costs. Often, we effectively use foundation models as feature generators and train classifiers that use the outputs of these models to make decisions. In this paper, we consider an increasingly relevant setting where we have two classifier stages. The first stage has access to features $x$ and has the option to make a classification decision or defer, while incurring a cost, to a second classifier that has access to features $x$ and $z$. This is similar to the ``learning to defer'' setting, with the important difference that we train both classifiers jointly, and the second classifier has access to more information. The natural loss for this setting is an $\ell_{01c}$ loss, where a penalty is paid for incorrect classification, as in $\ell_{01}$, but an additional penalty $c$ is paid for consulting the second classifier. The $\ell_{01c}$ loss is unwieldy for training. Our primary contribution in this paper is the derivation of a hinge-based surrogate loss $\ell^c_{hinge}$ that is much more amenable to training but also satisfies the property that $\ell^c_{hinge}$-consistency implies $\ell_{01c}$-consistency.

Large language models (LLMs) have achieved impressive results on complex reasoning tasks, but their high inference cost remains a major barrier to real-world deployment. A promising solution is to use cascaded inference, where small, cheap models handle easy queries, and only the hardest examples are escalated to more powerful models. However, existing cascade methods typically rely on supervised training with labeled data, offer no theoretical generalization guarantees, and provide limited control over test-time computational cost. We introduce **C3PO** (*Cost Controlled Cascaded Prediction Optimization*), a self-supervised framework for optimizing LLM cascades under probabilistic cost constraints. By focusing on minimizing regret with respect to the most powerful model (MPM), C3PO avoids the need for labeled data by constructing a cascade using only unlabeled model outputs. It leverages conformal prediction to bound the probability that inference cost exceeds a user-specified budget. We provide theoretical guarantees on both cost control and generalization error, and show that our optimization procedure is effective even with small calibration sets. Empirically, C3PO achieves state-of-the-art performance across a diverse set of reasoning benchmarks including GSM8K, MATH-500, BigBench-Hard and AIME, outperforming strong LLM cascading baselines in both accuracy and cost-efficiency. Our results demonstrate that principled, label-free cascade optimization can enable scalable LLM deployment.

As we expect to see a rapid integration of generative models in our day to day lives, the development of rigorous methods of evaluation and analysis for generative models has never been more pressing. Multiple works have highlighted the shortcomings of widely used metrics and exposed how they fail to behave as expected in some settings. So far, the response has been to use a variety of metrics that target different desirable and interpretable properties such as fidelity, diversity, and authenticity, to obtain a clearer picture of a generative model’s capabilities. These methods mainly focus on ordinal data and they all suffer from the same unavoidable issues stemming from estimating quantities of high-dimensional data from a limited number of samples. We propose to take an alternative approach and to return to the synthetic data setting where the ground truth is explicit and known. We focus on nominal categorical data and introduce an evaluation method that can scale to the high-dimensional settings often encountered in practice. Our method involves successively binning the large space to obtain smaller probability spaces and coarser distributions where meaningful statistical estimates can be obtained. This allows us to provide probabilistic guarantees and sample complexities and we illustrate how our method can be applied to distinguish between the capabilities of several state-of-the-art categorical models.

The performance of transformers for time-series forecasting has improved significantly. Recent architectures learn complex temporal patterns by segmenting a time-series into patches and using the patches as tokens. The patch size controls the ability of transformers to learn the temporal patterns at different frequencies: shorter patches are effective for learning localized, high-frequency patterns, whereas mining long-term seasonalities and trends requires longer patches. Inspired by this observation, we propose a novel framework, Multi-resolution Time-Series Transformer (MTST), which consists of a multi-branch architecture for simultaneous modeling of diverse temporal patterns at different resolutions. In contrast to many existing time-series transformers, we employ relative positional encoding, which is better suited for extracting periodic components at different scales. Extensive experiments on several real-world datasets demonstrate the effectiveness of MTST in comparison to state-of-the-art forecasting techniques.

Many platforms, such as e-commerce websites, offer both search and recommendation services simultaneously to better meet users' diverse needs. Recommendation services suggest items based on user preferences, while search services allow users to search for items before providing recommendations. Since users and items are often shared between the search and recommendation domains, there is a valuable opportunity to enhance the recommendation domain by leveraging user preferences extracted from the search domain. Existing approaches either overlook the shift in user intention between these domains or fail to capture the significant impact of learning from users' search queries on understanding their interests.

Large pretrained models, coupled with fine-tuning, are slowly becoming established as the dominant architecture in machine learning. Even though these models offer impressive performance, their practical application is often limited by the prohibitive amount of resources required for $\textit{every}$ inference. Early-exiting dynamic neural networks (EDNN) circumvent this issue by allowing a model to make some of its predictions from intermediate layers (i.e., early-exit). Training an EDNN architecture is challenging as it consists of two intertwined components: the gating mechanism (GM) that controls early-exiting decisions and the intermediate inference modules (IMs) that perform inference from intermediate representations. As a result, most existing approaches rely on thresholding confidence metrics for the gating mechanism and strive to improve the underlying backbone network and the inference modules. Although successful, this approach has two fundamental shortcomings: 1) the GMs and the IMs are decoupled during training, leading to a train-test mismatch; and 2) the thresholding gating mechanism introduces a positive bias into the predictive probabilities, making it difficult to readily extract uncertainty information. We propose a novel architecture that connects these two modules. This leads to significant performance improvements on classification datasets and enables better uncertainty characterization capabilities.

The existing definitions of graph convolution, either from spatial or spectral perspectives, are inflexible and not unified. Defining a general convolution operator in the graph domain is challenging due to the lack of canonical coordinates, the presence of irregular structures, and the properties of graph symmetries. In this work, we propose a novel and general graph convolution framework by parameterizing the kernels as continuous functions of pseudo-coordinates derived via graph positional encoding. We name this Continuous Kernel Graph Convolution (CKGConv). Theoretically, we demonstrate that CKGConv is flexible and expressive. CKGConv encompasses many existing graph convolutions, and exhibits a stronger expressiveness, as powerful as graph transformers in terms of distinguishing non-isomorphic graphs. Empirically, we show that CKGConv-based Networks outperform existing graph convolutional networks and perform comparably to the best graph transformers across a variety of graph datasets. The code and models are publicly available at https://github.com/networkslab/CKGConv.

Neural Temporal Point Processes (TPPs) have emerged as the primary framework for predicting sequences of events that occur at irregular time intervals, but their sequential nature can hamper performance for long-horizon forecasts. To address this, we introduce a novel approach that incorporates a diffusion generative model. The model facilitates sequence-to-sequence prediction, allowing multi-step predictions based on historical event sequences. In contrast to previous approaches, our model directly learns the joint probability distribution of types and inter-arrival times for multiple events. The model is composed of two diffusion processes, one for the time intervals and one for the event types. These processes interact through their respective denoising functions, which can take as input intermediate representations from both processes, allowing the model to learn complex interactions. We demonstrate that our proposal outperforms state-of-the-art baselines for long-horizon forecasting of TPPs.

Boolean satisfiability (SAT) problems are routinely solved by SAT solvers in real-life applications, yet solving time can vary drastically between solvers for the same instance.This has motivated research into machine learning models that can predict, for a given SAT instance, which solver to select among several options.Existing SAT solver selection methods all rely on some hand-picked instance features, which are costly to compute and ignore the structural information in SAT graphs.In this paper we present GraSS, a novel approach for automatic SAT solver selection based on tripartite graph representations of instances and a heterogeneous graph neural network (GNN) model.While GNNs have been previously adopted in other SAT-related tasks, they do not incorporate any domain-specific knowledge and ignore the runtime variation introduced by different clause orders.We enrich the graph representation with domain-specific decisions, such as novel node feature design, positional encodings for clauses in the graph, a GNN architecture tailored to our tripartite graphs and a runtime-sensitive loss function.Through extensive experiments, we demonstrate that this combination of raw representations and domain-specific choices leads to improvements in runtime for a pool of seven state-of-theart solvers on both an industrial circuit design benchmark, and

Efficiently determining the satisfiability of a boolean equation --- known as the SAT problem for brevity --- is crucial in various industrial problems. Recently, the advent of deep learning methods has introduced significant potential for enhancing SAT solving. However, a major barrier to the advancement of this field has been the scarcity of large, realistic datasets. The majority of current public datasets are either randomly generated or extremely limited, containing only a few examples from unrelated problem families. These datasets are inadequate for meaningful training of deep learning methods. In light of this, researchers have started exploring generative techniques to create data that more accurately reflect SAT problems encountered in practical situations. These methods have so far suffered from either the inability to produce challenging SAT problems or time-scalability obstacles. In this paper we address both by identifying and manipulating the key contributors to a problem's ``hardness'', known as cores. Although some previous work has addressed cores, the time costs are unacceptably high due to the expense of traditional heuristic core detection techniques. We introduce a fast core detection procedure that uses a graph neural network. Our empirical results demonstrate that we can efficiently generate problems that remain hard to solve and retain key attributes of the original example problems. We show via experiment that the generated synthetic SAT problems can be used in a data augmentation setting to provide improved prediction of solver runtimes.

Adaptive importance sampling (AIS) methods provide a useful alternative to Markov Chain Monte Carlo (MCMC) algorithms for performing inference of intractable distributions. Population Monte Carlo (PMC) algorithms constitute a family of AIS approaches which adapt the proposal distributions iteratively to improve the approximation of the target distribution. Recent work in this area primarily focuses on ameliorating the proposal adaptation procedure for high-dimensional applications. However, most of the AIS algorithms use simple proposal distributions for sampling, which might be inadequate in exploring target distributions with intricate geometries. In this work, we construct expressive proposal distributions in the AIS framework using normalizing flow, an appealing approach for modeling complex distributions. We use an iterative parameter update rule to enhance the approximation of the target distribution. Numerical experiments show that in high-dimensional settings, the proposed algorithm offers significantly improved performance compared to the existing techniques.

We show that the task of synthesizing human motion conditioned on a set of key frames can be solved more accurately and effectively if a deep learning based interpolator operates in the delta mode using the spherical linear interpolator as a baseline. We empirically demonstrate the strength of our approach on publicly available datasets achieving state-of-the-art performance. We further generalize these results by showing that the ∆-regime is viable with respect to the reference of the last known frame (also known as the zero-velocity model). This supports the more general conclusion that operating in the reference frame local to input frames is more accurate and robust than in the global (world) reference frame advocated in previous work.

The mobile communication enabled by cellular networks is the one of the main foundations of our modern society. Optimizing the performance of cellular networks and providing massive connectivity with improved coverage and user experience has a considerable social and economic impact on our daily life. This performance relies heavily on the configuration of the network parameters. However, with the massive increase in both the size and complexity of cellular networks, network management, especially parameter configuration, is becoming complicated. The current practice, which relies largely on experts' prior knowledge, is not adequate and will require lots of domain experts and high maintenance costs. In this work, we propose a learning-based framework for handover parameter configuration. The key challenge, in this case, is to tackle the complicated dependencies between neighboring cells and jointly optimize the whole network. Our framework addresses this challenge in two ways. First, we introduce a novel approach to imitate how the network responds to different network states and parameter values, called auto-grouping graph convolutional network (AG-GCN). During the parameter configuration stage, instead of solving the global optimization problem, we design a local multi-objective optimization strategy where each cell considers several local performance metrics to balance its own performance and its neighbors. We evaluate our proposed algorithm via a simulator constructed using real network data. We demonstrate that the handover parameters our model can find, achieve better average network throughput compared to those recommended by experts as well as alternative baselines, which can bring better network quality and stability. It has the potential to massively reduce costs arising from human expert intervention and maintenance.

Learning a categorical distribution comes with its own set of challenges. A successful approach taken by state-of-the-art works is to cast the problem in a continuous domain to take advantage of the impressive performance of the generative models for continuous data. Amongst them are the recently emerging diffusion probabilistic models, which have the observed advantage of generating high-quality samples. Recent advances for categorical generative models have focused on log likelihood improvements. In this work, we propose a generative model for categorical data based on diffusion models with a focus on high-quality sample generation, and propose sampled-based evaluation methods. The efficacy of our method stems from performing diffusion in the continuous domain while having its parameterization informed by the structure of the categorical nature of the target distribution. Our method of evaluation highlights the capabilities and limitations of different generative models for generating categorical data, and includes experiments on synthetic and real-world protein datasets.

Recommender systems now consume large-scale data and play a significant role in improving user experience. Graph Neural Networks (GNNs) have emerged as one of the most effective recommender system models because they model the rich relational information. The ever-growing volume of data can make training GNNs prohibitively expensive. To address this, previous attempts propose to train the GNN models incrementally as new data blocks arrive. Feature and structure knowledge distillation techniques have been explored to allow the GNN model to train in a fast incremental fashion while alleviating the catastrophic forgetting problem. However, preserving the same amount of the historical information for all users is sub-optimal since it fails to take into account the dynamics of each user's change of preferences. For the users whose interests shift substantially, retaining too much of the old knowledge can overly constrain the model, preventing it from quickly adapting to the users’ novel interests. In contrast, for users who have static preferences, model performance can benefit greatly from preserving as much of the user's long-term preferences as possible. In this work, we propose a novel training strategy that adaptively learns personalized imitation weights for each user to balance the contribution from the recent data and the amount of knowledge to be distilled from previous time periods. We demonstrate the effectiveness of learning imitation weights via a comparison on five diverse datasets for three state-of-art structure distillation based recommender systems. The performance shows consistent improvement over competitive incremental learning techniques.

Contrastive learning has emerged as a premier method for learning representations with or without supervision. Recent studies have shown its utility in graph representation learning for pre-training. Despite successes, the understanding of how to design effective graph augmentations that can capture structural properties common to many different types of downstream graphs remains incomplete. We propose a set of well-motivated graph transformation operations derived via graph spectral analysis to provide a bank of candidates when constructing augmentations for a graph contrastive objective, enabling contrastive learning to capture useful structural representation from pre-training graph datasets. We first present a spectral graph cropping augmentation that involves filtering nodes by applying thresholds to the eigenvalues of the leading Laplacian eigenvectors. Our second novel augmentation reorders the graph frequency components in a structural Laplacian-derived position graph embedding. Further, we introduce a method that leads to improved views of local subgraphs by performing alignment via global random walk embeddings. Our experimental results indicate consistent improvements in out-of-domain graph data transfer compared to state-of-the-art graph contrastive learning methods, shedding light on how to design a graph learner that is able to learn structural properties common to diverse graph types.

The machine learning community has mainly relied on real data to benchmark algorithms as it provides compelling evidence of model applicability. Evaluation on synthetic datasets can be a powerful tool to provide a better understanding of a model’s strengths, weaknesses and overall capabilities. Gaining these insights can be particularly important for generative modeling as the target quantity is completely unknown. Multiple issues related to the evaluation of generative models have been reported in the literature. We argue those problems can be avoided by an evaluation based on ground truth. General criticisms of synthetic experiments are that they are too simplified and not representative of practical scenarios. As such, our experimental setting is tailored to a realistic generative task. We focus on categorical data and introduce an appropriately scalable evaluation method. Our method involves tasking a generative model to learn a distribution in a high-dimensional setting. We then successively bin the large space to obtain smaller probability spaces where meaningful statistical tests can be applied. We consider increasingly large probability spaces, which correspond to increasingly difficult modeling tasks, and compare the generative models based on the highest task difficulty they can reach before being detected as being too far from the ground truth. We validate our evaluation procedure with synthetic experiments on both synthetic generative models and current state-of-the-art categorical generative models.

One of the most crucial concerns in Neural Architecture Search (NAS) is the amount of resources and time required for the search. In this paper we address the problem of training-free (or zero-shot) search. We propose a generalized architecture search framework called enhanced training-free neural architecture search (ETE-NAS). It ranks architectures according to a user defined combination of zero-shot ranking functions. We instantiate it with metrics based on neural network Gaussian process (NNGP) kernel and ReLU regions division. This allows much faster search on a single GPU than other training-free approaches without sacrificing accuracy or inducing instability. Code is available at https://github.com/Rufaim/ETENAS

To offer accurate and diverse recommendation services, recent methods use auxiliary information to foster the learning process of user and item representations. Many state-of-the-art (SOTA) methods fuse different sources of information (user, item, knowledge graph, tags, etc.) into a graph and use Graph Neural Networks (GNNs) to introduce the auxiliary information through the message passing paradigm. In this work, we seek an alternative framework that is light and effective through self-supervised learning across different sources of information, particularly for the commonly accessible item tag information. We use a self-supervision signal to pair users with the auxiliary information (tags) associated with the items they have interacted with before. To achieve the pairing, we create a proxy training task. For a given item, the model predicts which is the correct pairing between the representations obtained from the users that have interacted with this item and the tags assigned to it. This design provides an efficient solution, using the auxiliary information directly to enhance the quality of user and item embeddings. User behavior in recommendation systems is driven by the complex interactions of many factors behind the users’ decision-making processes. To make the pairing process more fine-grained and avoid embedding collapse, we propose a user intent-aware self-supervised pairing process where we split the user embeddings into multiple sub-embedding vectors. Each sub-embedding vector captures a specific user intent via self-supervised alignment with a particular cluster of tags. We integrate our designed framework with various recommendation models, demonstrating its flexibility and compatibility. Through comparison with numerous SOTA methods on seven real-world datasets, we show that our method can achieve better performance while requiring less training time. This indicates the potential of applying our approach on web-scale datasets.

Offline model-based optimization aims to maximize a black-box objective function with a static dataset of designs and their scores. In this paper, we focus on biological sequence design to maximize some sequence score. A recent approach employs bidirectional learning, combining a forward mapping for exploitation and a backward mapping for constraint, and it relies on the neural tangent kernel (NTK) of an infinitely wide network to build a proxy model. Though effective, the NTK cannot learn features because of its parametrization, and its use prevents the incorporation of powerful pre-trained Language Models (LMs) that can capture the rich biophysical information in millions of biological sequences. We adopt an alternative proxy model, adding a linear head to a pre-trained LM, and propose a linearization scheme. This yields a closed-form loss and also takes into account the biophysical information in the pre-trained LM. In addition, the forward mapping and the backward mapping play different roles and thus deserve different weights during sequence optimization. To achieve this, we train an auxiliary model and leverage its weak supervision signal via a bi-level optimization framework to effectively learn how to balance the two mappings. Further, by extending the framework, we develop the first learning rate adaptation module Adaptive-$\eta$, which is compatible with all gradient-based algorithms for offline model-based optimization. Experimental results on DNA/protein sequence design tasks verify the effectiveness of our algorithm. Our code is available at https://github.com/GGchen1997/BIB-ICML2023-Submission.

Transformers for graph data are increasingly widely studied and successful in numerous learning tasks. Graph inductive biases are crucial for Graph Transformers, and previous works incorporate them using message-passing modules and/or positional encodings. However, Graph Transformers that use message-passing inherit known issues of message-passing, and differ significantly from Transformers used in other domains, thus making transfer of research advances more difficult. On the other hand, Graph Transformers without message-passing often perform poorly on smaller datasets, where inductive biases are more crucial. To bridge this gap, we propose the Graph Inductive bias Transformer (GRIT) --- a new Graph Transformer that incorporates graph inductive biases without using message passing. GRIT is based on several architectural changes that are each theoretically and empirically justified, including: learned relative positional encodings initialized with random walk probabilities, a flexible attention mechanism that updates node and node-pair representations, and injection of degree information in each layer. We prove that GRIT is expressive --- it can express shortest path distances and various graph propagation matrices. GRIT achieves state-of-the-art empirical performance across a variety of graph datasets, thus showing the power that Graph Transformers without message-passing can deliver.

We describe a new setting for learning a deep graph generative model (GGM) from aggregate graph statistics, rather than from the graph adjacency matrix. Matching the statistics of observed training graphs is the main approach for learning traditional GGMs (e.g, BTER, Chung-Lu, and Erdos-Renyi models). Privacy researchers have proposed learning from graph statistics as a way to protect privacy. We develop an architecture for training a deep GGM to match statistics while preserving local differential privacy guarantees. Empirical evaluation on 8 datasets indicates that our deep GGM model generates more realistic graphs than the traditional GGMs when both are learned from graph statistics only. We also benchmark our deep GGM trained on statistics only, against state-of-the-art deep GGM models that are trained on the entire adjacency matrix. The results show that graph statistics are often sufficient to build a competitive deep GGM that generates realistic graphs while protecting local privacy.

Multiple Instance Learning (MIL) is a weakly supervised learning problem where the aim is to assign labels to sets or bags of instances, as opposed to traditional supervised learning where each instance is assumed to be independent and identically distributed (IID) and is to be labeled individually. Recent work has shown promising results for neural network models in the MIL setting. Instead of focusing on each instance, these models are trained in an end-to-end fashion to learn effective bag-level representations by suitably combining permutation invariant pooling techniques with neural architectures. In this paper, we consider modelling the interactions between bags using a graph and employ Graph Neural Networks (GNNs) to facilitate end-to-end learning. Since a meaningful graph representing dependencies between bags is rarely available, we propose to use a Bayesian GNN framework that can generate a likely graph structure for scenarios where there is uncertainty in the graph or when no graph is available. Empirical results demonstrate the efficacy of the proposed technique for several MIL benchmark tasks and a distribution regression task.

Implicit feedback is frequently used for developing personalized recommendation services due to its ubiquity and accessibility in real-world systems. In order to effectively utilize such information, most research adopts the pairwise ranking method on constructed training triplets (user, positive item, negative item) and aims to distinguish between positive items and negative items for each user. However, most of these methods treat all the training triplets equally, which ignores the subtle difference between different positive or negative items. On the other hand, even though some other works make use of the auxiliary information (e.g., dwell time) of user behaviors to capture this subtle difference, such auxiliary information is hard to obtain. To mitigate the aforementioned problems, we propose a novel training framework named Triplet Importance Learning (TIL), which adaptively learns the importance score of training triplets. We devise two strategies for the importance score generation and formulate the whole procedure as a bilevel optimization, which does not require any rule-based design. We integrate the proposed training procedure with several Matrix Factorization (MF)- and Graph Neural Network (GNN)-based recommendation models, demonstrating the compatibility of our framework. Via a comparison using three real-world datasets with many state-of-the-art methods, we show that our proposed method outperforms the best existing models by 3-21% in terms of [email protected] for the top-k recommendation.

A well-known challenge associated with the multi-label classification problem is modelling dependencies between labels. Most attempts at modelling label dependencies focus on co-occurrences, ignoring the valuable information that can be extracted by detecting label subsets that rarely occur together. For example, consider customer product reviews; a product probably would not simultaneously be tagged by both "recommended" (i.e., reviewer is happy and recommends the product) and "urgent" (i.e., the review suggests immediate action to remedy an unsatisfactory experience). Aside from the consideration of positive and negative dependencies, the direction of a relationship should also be considered. For a multi-label image classification problem, the "ship" and "sea" labels have an obvious dependency, but the presence of the former implies the latter much more strongly than the other way around. These examples motivate the modelling of multiple types of bi-directional relationships between labels. In this paper, we propose a novel method, entitled Multi-relation Message Passing (MrMP), for the multi-label classification problem. Experiments on benchmark multi-label text classification datasets show that the MrMP module yields similar or superior performance compared to state-of-the-art methods. The approach imposes only minor additional computational and memory overheads.1

In offline model-based optimization, we strive to maximize a black-box objective function by only leveraging a static dataset of designs and their scores. This problem setting arises in numerous fields including the design of materials, robots, DNAs, proteins, etc. Recent approaches train a deep neural network (DNN) model on the static dataset to act as a proxy function, and then perform gradient ascent on the existing designs to obtain potentially high-scoring designs. This methodology frequently suffers from the out-of-distribution problem where the proxy function often returns adversarial designs. To mitigate this problem, we propose $\textit{\textbf{B}i\textbf{D}irectional learning for offline \textbf{I}nfinite-width model-based optimization}~(\textbf{BDI})$. BDI consists of two mappings: the forward mapping leverages the static dataset to predict the scores of the high-scoring designs, and the backward mapping leverages the high-scoring designs to predict the scores of the static dataset. The backward mapping, neglected in previous work, can distill more information of the static dataset into the high-scoring designs, which effectively mitigates the out-of-distribution problem. Yet, for a finite-width DNN model, the loss function of the backward mapping is intractable and only has an approximate form, which leads to a significant deterioration of the design quality. We thus adopt an infinite-width DNN model and propose to employ the corresponding neural tangent kernel to yield a closed-form loss for more accurate design updates. Experiments on various tasks verify the effectiveness of BDI. The code is available [here](https://github.com/GGchen1997/BDI).

Personalized recommender systems are increasingly important as more content and services become available and users struggle to identify what might interest them. Thanks to the ability for providing rich information, knowledge graphs (KGs) are being incorporated to enhance the recommendation performance and interpretability. To effectively make use of the knowledge graph, we propose a recommendation model in the hyperbolic space, which facilitates the learning of the hierarchical structure of knowledge graphs. Furthermore, a hyperbolic attention network is employed to determine the relative importances of neighboring entities of a certain item. In addition, we propose an adaptive and fine-grained regularization mechanism to adaptively regularize items and their neighboring representations. Via a comparison using three real-world datasets with state-of-the-art methods, we show that the proposed model outperforms the best existing models by 2-16% in terms of NDCG@K on Top-K recommendation.

Forecasting of multivariate time-series is an important problem that has applications in traffic management, cellular network configuration, and quantitative finance. A special case of the problem arises when there is a graph available that captures the relationships between the time-series. In this paper we propose a novel learning architecture that achieves performance competitive with or better than the best existing algorithms, without requiring knowledge of the graph. The key element of our proposed architecture is the learnable fully connected hard graph gating mechanism that enables the use of the state-of-the-art and highly computationally efficient fully connected time-series forecasting architecture in traffic forecasting applications. Experimental results for two public traffic network datasets illustrate the value of our approach, and ablation studies confirm the importance of each element of the architecture. The code is available here: https://github.com/boreshkinai/fc-gaga.

Graph neural network (GNN) models achieve superior performance when classifying nodes in graph-structured data. Given that state-of-the-art GNNs share many similarities with their CNN cousins and that CNNs suffer adversarial vulnerabilities, there has also been interest in exploring analogous vulnerabilities in GNNs. Indeed, recent work has demonstrated that node classification performance of several graph models, including the popular graph convolution network (GCN) model, can be severely degraded through adversarial perturbations to the graph structure and the node features. In this work, we take a first step towards detecting adversarial attacks against graph models. We first propose a straightforward single node threshold test for detecting nodes subject to targeted attacks. Subsequently, we describe a kernel-based two-sample test for detecting whether a given subset of nodes within a graph has been maliciously corrupted. The efficacy of our algorithms is established via thorough experiments using commonly used node classification benchmark datasets. We also illustrate the potential practical benefit of our detection method by demonstrating its application to a real-world Bitcoin transaction network.

Large-scale recommender systems are integral parts of many services. With the recent rapid growth of accessible data, the need for efficient training methods has arisen. Given the high computational cost of training state-of-the-art graph neural network (GNN) based models, it is infeasible to train them from scratch with every new set of interactions. In this work, we present a novel framework for incrementally training GNN-based models. Our framework takes advantage of an experience reply technique built on top of a structurally aware reservoir sampling method tailored for this setting. This framework addresses catastrophic forgetting, allowing the model to preserve its understanding of users' long-term behavioral patterns while adapting to new trends. Our experiments demonstrate the superior performance of our framework on numerous datasets when combined with state-of-the-art GNN-based models.

Personalized recommender systems are playing an increasingly important role for online services. Graph Neural Network (GNN) based recommender models have demonstrated a superior capability to model users' interests thanks to rich relational information encoded in graphs. However, with the ever-growing volume of online information and the high computational complexity of training GNNs, it is difficult to perform frequent updates to provide the most up-to-date recommendations. There have been several attempts towards training GNN models in an incremental fashion to enable faster training times and permit more frequent model updates using the latest training data. The main technique is knowledge distillation, which aims to allow model updates while preserving key aspects of the model that were learned from the historical data. In this work, we develop a novel Graph Structure Aware Contrastive Knowledge Distillation for Incremental Learning in recommender systems, which is tailored to focus on the rich relational information in the recommendation context. We combine the contrastive distillation formulation with intermediate layer distillation to inject layer-level supervision. We demonstrate the effectiveness of our proposed distillation framework for GNN based recommendation systems on four commonly used datasets, showing consistent improvement over state-of-the-art alternatives.

Presently with technology node scaling, an accurate prediction model at early design stages can significantly reduce the design cycle. Especially during logic synthesis, predicting cell congestion due to improper logic combination can reduce the burden of subsequent physical implementations. There have been attempts using Graph Neural Network (GNN) techniques to tackle congestion prediction during the logic synthesis stage. However, they require informative cell features to achieve reasonable performance since the core idea of GNNs is built on the message passing framework, which would be impractical at the early logic synthesis stage. To address this limitation, we propose a framework that can directly learn embeddings for the given netlist to enhance the quality of our node features. Popular random-walk based embedding methods such as Node2vec, LINE, and DeepWalk suffer from the issue of cross-graph alignment and poor generalization to unseen netlist graphs, yielding inferior performance and costing significant runtime. In our framework, we introduce a superior alternative to obtain node embeddings that can generalize across netlist graphs using matrix factorization methods. We propose an efficient mini-batch training method at the sub-graph level that can guarantee parallel training and satisfy the memory restriction for large-scale netlists. We present results utilizing open-source EDA tools such as DREAMPLACE and OPENROAD frameworks on a variety of openly available circuits. By combining the learned embedding on top of the netlist with the GNNs, our method improves prediction performance, generalizes to new circuit lines, and is efficient in training, potentially saving over 90% of runtime.

Spatio-temporal forecasting has numerous applications in analyzing wireless, traffic, and financial networks. Many classical statistical models often fall short in handling the complexity and high non-linearity present in time-series data. Recent advances in deep learning allow for better modelling of spatial and temporal dependencies. While most of these models focus on obtaining accurate point forecasts, they do not characterize the prediction uncertainty. In this work, we consider the time-series data as a random realization from a nonlinear state-space model and target Bayesian inference of the hidden states for probabilistic forecasting. We use particle flow as the tool for approximating the posterior distribution of the states, as it is shown to be highly effective in complex, high-dimensional settings. Thorough experimentation on several real world time-series datasets demonstrates that our approach provides better characterization of uncertainty while maintaining comparable accuracy to the state-of-the-art point forecasting methods.

Reasoning in a temporal knowledge graph (TKG) is a critical task for information retrieval and semantic search. It is particularly challenging when the TKG is updated frequently. The model has to adapt to changes in the TKG for efficient training and inference while preserving its performance on historical knowledge. Recent work approaches TKG completion (TKGC) by augmenting the encoder-decoder framework with a time-aware encoding function. However, naively fine-tuning the model at every time step using these methods does not address the problems of 1) catastrophic forgetting, 2) the model's inability to identify the change of facts (e.g., the change of the political affiliation and end of a marriage), and 3) the lack of training efficiency. To address these challenges, we present the Time-aware Incremental Embedding (TIE) framework, which combines TKG representation learning, experience replay, and temporal regularization. We introduce a set of metrics that characterizes the intransigence of the model and propose a constraint that associates the deleted facts with negative labels.

The chronological order of user-item interactions can reveal time-evolving and sequential user behaviors in many recommender systems. The items that users will interact with may depend on the items accessed in the past. However, the substantial increase of users and items makes sequential recommender systems still face non-trivial challenges: (1) the hardness of modeling the short-term user interests; (2) the difficulty of capturing the long-term user interests; (3) the effective modeling of item co-occurrence patterns. To tackle these challenges, we propose a memory augmented graph neural network (MA-GNN) to capture both the long- and short-term user interests. Specifically, we apply a graph neural network to model the item contextual information within a short-term period and utilize a shared memory network to capture the long-range dependencies between items. In addition to the modeling of user interests, we employ a bilinear function to capture the co-occurrence patterns of related items. We extensively evaluate our model on five real-world datasets, comparing with several state-of-the-art methods and using a variety of performance metrics. The experimental results demonstrate the effectiveness of our model for the task of Top-K sequential recommendation.

Given the convenience of collecting information through online services, recommender systems now consume large scale data and play a more important role in improving user experience. With the recent emergence of Graph Neural Networks (GNNs), GNN-based recommender models have shown the advantage of modeling the recommender system as a user-item bipartite graph to learn representations of users and items. However, such models are expensive to train and difficult to perform frequent updates to provide the most up-to-date recommendations. In this work, we propose to update GNN-based recommender models incrementally so that the computation time can be greatly reduced and models can be updated more frequently. We develop a Graph Structure Aware Incremental Learning framework, GraphSAIL, to address the commonly experienced catastrophic forgetting problem that occurs when training a model in an incremental fashion. Our approach preserves a user's long-term preference (or an item's long-term property) during incremental model updating. GraphSAIL implements a graph structure preservation strategy which explicitly preserves each node's local structure, global structure, and self-information, respectively. We argue that our incremental training framework is the first attempt tailored for GNN based recommender systems and demonstrate its improvement compared to other incremental learning techniques on two public datasets. We further verify the effectiveness of our framework on a large-scale industrial dataset.

Node classification in attributed graphs is an important task in multiple practical settings, but it can often be difficult or expensive to obtain labels. Active learning can improve the achieved classification performance for a given budget on the number of queried labels. The best existing methods are based on graph neural networks, but they often perform poorly unless a sizeable validation set of labelled nodes is available in order to choose good hyperparameters. We propose a novel graph-based active learning algorithm for the task of node classification in attributed graphs; our algorithm uses graph cognizant logistic regression, equivalent to a linearized graph-convolutional neural network (GCN), for the prediction phase and maximizes the expected error reduction in the query phase. To reduce the delay experienced by a labeller interacting with the system, we derive a preemptive querying system that calculates a new query during the labelling process, and to address the setting where learning starts with almost no labelled data, we also develop a hybrid algorithm that performs adaptive model averaging of label propagation and linearized GCN inference. We conduct experiments on five public benchmark datasets, demonstrating a significant improvement over state-of-the-art approaches and illustrate the practical value of the method by applying it to a private microwave link network dataset.

Personalized recommender systems are playing an increasingly important role as more content and services become available and users struggle to identify what might interest them. Although matrix factorization and deep learning based methods have proved effective in user preference modeling, they violate the triangle inequality and fail to capture fine-grained preference information. To tackle this, we develop a distance-based recommendation model with several novel aspects: (i) each user and item are parameterized by Gaussian distributions to capture the learning uncertainties; (ii) an adaptive margin generation scheme is proposed to generate the margins regarding different training triplets; (iii) explicit user-user/item-item similarity modeling is incorporated in the objective function. The Wasserstein distance is employed to determine preferences because it obeys the triangle inequality and can measure the distance between probabilistic distributions. Via a comparison using five real-world datasets with state-of-the-art methods, the proposed model outperforms the best existing models by 4-22% in terms of [email protected] on Top-K recommendation.

Personalized recommender systems are playing an increasingly important role for online consumption platforms. Because of the multitude of relationships existing in recommender systems, Graph Neural Networks (GNNs) based approaches have been proposed to better characterize the various relationships between a user and items while modeling a user's preferences. Previous graph-based recommendation approaches process the observed user-item interaction graph as a ground-truth depiction of the relationships between users and items. However, especially in the implicit recommendation setting, all the unobserved user-item interactions are usually assumed to be negative samples. There are missing links that represent a user's future actions. In addition, there may be spurious or misleading positive interactions. To alleviate the above issue, in this work, we take a first step to introduce a principled way to model the uncertainty in the user-item interaction graph using the Bayesian Graph Convolutional Neural Network framework. We discuss how inference can be performed under our framework and provide a concrete formulation using the Bayesian Probabilistic Ranking training loss. We demonstrate the effectiveness of our proposed framework on four benchmark recommendation datasets. The proposed method outperforms state-of-the-art graph-based recommendation models. Furthermore, we conducted an offline evaluation on one industrial large-scale dataset. It shows that our proposed method outperforms the baselines, with the potential gain being more significant for cold-start users. This illustrates the potential practical benefit in real-world recommender systems.

Personalized recommendation plays an important role in many online services. Substantial research has been dedicated to learning embeddings of users and items to predict a user's preference for an item based on the similarity of the representations. In many settings, there is abundant relationship information, including user-item interaction history, user-user and item-item similarities. In an attempt to exploit these relationships to learn better embeddings, researchers have turned to the emerging field of Graph Convolutional Neural Networks (GCNs), and applied GCNs for recommendation. Although these prior works have demonstrated promising performance, directly apply GCNs to process the user-item bipartite graph is suboptimal because the GCNs do not consider the intrinsic differences between user nodes and item nodes. Additionally, existing large-scale graph neural networks use aggregation functions such as sum/mean/max pooling operations to generate a node embedding that considers the nodes' neighborhood (i.e., the adjacent nodes in the graph), and these simple aggregation strategies fail to preserve the relational information in the neighborhood. To resolve the above limitations, in this paper, we propose a novel framework NIA-GCN, which can explicitly model the relational information between neighbor nodes and exploit the heterogeneous nature of the user-item bipartite graph. We conduct empirical studies on four public benchmarks, demonstrating a significant improvement over state-of-the-art approaches. Furthermore, we generalize our framework to a commercial App store recommendation scenario. We observe significant improvement on a large-scale commercial dataset, demonstrating the practical potential for our proposed solution as a key component of a large scale commercial recommender system. Furthermore, online experiments are conducted to demonstrate that NIA-GCN outperforms the baseline by 10.19% and 9.95% in average in terms of CTR and CVR during ten-day AB test in a mainstream App store.

It is recognized that system trustworthiness is a hyperstructure embodied by the structural, behavioral, and system dimensions with a set of coherent attributes. We explore a theoretical framework of tripartite trustworthiness that can be applied to real-world autonomous systems. We present a formal study of the essences and mathematical models of system trustworthiness and their quantitative measurements in the contexts of autonomous and mission-critical intelligent systems where humans and machines interact in a hybrid environment.

Graphs are ubiquitous in modelling relationalstructures. Recent endeavours in machine learningfor graph structured data have led to manyarchitectures and learning algorithms. However,the graph used by these algorithms is oftenconstructed based on inaccurate modellingassumptions and/or noisy data. As a result, itfails to represent the true relationships betweennodes. A Bayesian framework which targetsposterior inference of the graph by consideringit as a random quantity can be beneficial. Inthis paper, we propose a novel non-parametricgraph model for constructing the posterior distributionof graph adjacency matrices. The proposedmodel is flexible in the sense that it caneffectively take into account the output of graphbased learning algorithms that target specifictasks. In addition, model inference scales wellto large graphs. We demonstrate the advantagesof this model in three different problem settings:node classification, link prediction andrecommendation.

Recently, techniques for applying convolutional neural networks to graph-structured data have emerged. Graph convolutional neural networks (GCNNs) have been used to address node and graph classification and matrix completion. Although the performance has been impressive, the current implementations have limited capability to incorporate uncertainty in the graph structure. Almost all GCNNs process a graph as though it is a ground-truth depiction of the relationship between nodes, but often the graphs employed in applications are themselves derived from noisy data or modelling assumptions. Spurious edges may be included; other edges may be missing between nodes that have very strong relationships. In this paper we adopt a Bayesian approach, viewing the observed graph as a realization from a parametric family of random graphs. We then target inference of the joint posterior of the random graph parameters and the node (or graph) labels. We present the Bayesian GCNN framework and develop an iterative learning procedure for the case of assortative mixed-membership stochastic block models. We present the results of experiments that demonstrate that the Bayesian formulation can provide better performance when there are very few labels available during the training process.

Community detection from graphs has many applications in machine learning, biological and social sciences. While there is a broad spectrum of literature based on various approaches, recently there has been a significant focus on inference algorithms for statistical models of community structure. These algorithms strive to solve an inference problem based on a generative model of the network. Recent advances in stochastic gradient MCMC have played a crucial role in improving the scalability of these techniques. In this paper, we propose a version of a degree corrected stochastic block model and present an MCMC based inference algorithm. Experimental results on several real world networks demonstrate the effectiveness of the proposed approach.

Improving disease outcome prediction can greatly aid in the strategic deployment of secondary prevention approaches. We develop a method to predict the evolution of diseases by taking into account personal attributes of the subjects and their relationships with medical examination results. Our approach builds upon a recent formulation of this problem as a graph-based geometric matrix completion task. The primary innovation is the introduction of multiple graphs, each relying on a different combination of subject attributes. Via statistical significance tests, we determine the relevant graph(s) for each medically-derived feature. We then employ a multiple-graph recurrent graph convolutional neural network architecture to predict the disease outcomes. We demonstrate the efficacy of the technique by addressing the task of predicting the development of Alzheimer's disease for patients exhibiting mild cognitive impairment, showing that the incorporation of multiple graphs improves predictive capability.

Personalized recommendation is ubiquitous, playing an important role in many online services. Substantial research has been dedicated to learning vector representations of users and items with the goal of predicting a user's preference for an item based on the similarity of the representations. Techniques range from classic matrix factorization to more recent deep learning based methods. However, we argue that existing methods do not make full use of the information that is available from user-item interaction data and the similarities between user pairs and item pairs. In this work, we develop a graph convolution-based recommendation framework, named Multi-Graph Convolution Collaborative Filtering (Multi-GCCF), which explicitly incorporates multiple graphs in the embedding learning process. Multi-GCCF not only expressively models the high-order information via a bipartite user-item interaction graph, but integrates the proximal information by building and processing user-user and item-item graphs. Furthermore, we consider the intrinsic difference between user nodes and item nodes when performing graph convolution on the bipartite graph. We conduct extensive experiments on four publicly accessible benchmarks, showing significant improvements relative to several state-of-the-art collaborative filtering and graph neural network-based recommendation models. Further experiments quantitatively verify the effectiveness of each component of our proposed model and demonstrate that the learned embeddings capture the important relationship structure.

Particle filters has become a standard tool for state estimation in nonlinear systems. However, their performance usually deteriorates if the dimension of state space is high or the measurements are highly informative. A major challenge is to construct a proposal density that is well matched to the posterior distribution. Particle flow methods are a promising option for addressing this task. In this paper, we develop a particle flow particle filter algorithm to address the case where both the process noise and the measurement noise are distributed as mixtures of Gaussians. Numerical experiments are performed to explore when the proposed method offers advantages compared to existing techniques.

Decentralized receding horizon control (D-RHC) provides a mechanism for coordination in multiagent settings without a centralized command center. However, combining a set of different goals, costs, and constraints to form an efficient optimization objective for D-RHC can be difficult. To allay this problem, we use a meta-learning process - cost adaptation - which generates the optimization objective for D-RHC to solve based on a set of human-generated priors (cost and constraint functions) and an auxiliary heuristic. We use this adaptive D-RHC method for control of mesh-networked swarm agents. This formulation allows a wide range of tasks to be encoded and can account for network delays, heterogeneous capabilities, and increasingly large swarms through the adaptation mechanism. We leverage the Unity3D game engine to build a simulator capable of introducing artificial networking failures and delays in the swarm. Using the simulator we validate our method on an example coordinated exploration task. We demonstrate that cost adaptation allows for more efficient and safer task completion under varying environment conditions and increasingly large swarm sizes. We release our simulator and code to the community for future work.

Particle filters are among the most effective filtering algorithms for nonlinear and non-Gaussian models. When the state dimension is high, they are known to suffer from weight degeneracy. Sequential Markov chain Monte Carlo (SMCMC) methods have been proposed as an alternative sequential inference technique that can perform better in high dimensional state spaces. In this paper, we propose to construct a composite Metropolis-Hastings (MH) kernel within the SMCMC framework using invertible particle flow. Simulation results show that the proposed kernel significantly increases the acceptance rate and improves estimation accuracy compared with state-of-the-art filtering algorithms, in high dimensional simulation examples.

In this paper we derive a particle flow particle filter implementation of the δ-Generalized Labeled Multi-Bernoulli (δ-GLMB) filter. The bootstrap particle filter δ-GLMB suffers from weight degeneracy for high-dimensional state systems or low measurement noise. In order to avoid weight degeneracy, we employ particle flow to produce a measurement-driven importance distribution that serves as a proposal in the δ-GLMB particle filter. Flow-induced proposals are developed for both types of targets encountered in the δ-GLMB filter, i.e., persistent and birth targets. Numerical simulations reflect the improved performance of the proposed filter with respect to classical bootstrap implementations.

Particle flow algorithms have been developed as an alternative to particle filtering. In these algorithms, there is no importance sampling, and particles are migrated from the prior to the posterior via a flow, described by differential equations. Aside from a few special cases, implementations involve multiple approximations, and their impact on the accuracy of the estimates is not clearly understood. In this paper, we propose algorithms that use particle flow procedures to construct an importance sampling distribution within a standard particle filter. The resultant algorithms retain the statistical consistency of sequential Monte Carlo methods, but acquire the desirable properties of particle flow techniques. We report the results of a multiple target tracking simulation study that combines highly informative measurements with a reasonably high-dimensional state space, leading to a challenging scenario for particle filters. Of the filters we test, the particle flow particle filter provides the smallest tracking error and achieves the largest average effective sample size.

We present a distributed cardinalized probability hypothesis density (CPHD) filter for multi-sensor multi-target tracking. Each sensor runs a single-sensor CPHD filter to compute the probability hypothesis density (PHD) function and cardinality distribution using only its own measurements and then fuses the local results by gossiping with neighboring sensors. Existing schemes that fuse local results using the Kullback-Leibler average are adversely affected if some sensors do not detect a target. The proposed fusion strategy, based on the arithmetic mean instead of the geometric mean, aims to be more robust to missed detections. We also show via simulations that the performance of the proposed algorithm can be significantly improved, with a small additional communication overhead, by having sensors exchange measurements locally.

We present a distributed particle filter for bearings-only tracking of a target moving on the surface of a sphere, such as Earth. The proposed filter accounts for the curvature of the surface in the measurement model for more robust performance. In addition, a linearization of the likelihood function significantly reduces the communication overhead. Simulations demonstrate that the proposed distributed approach maintains accuracy comparable to that of a centralized filter with access to all measurements even when the sensors and target are spread over a large region, where a planar approximation would fail.

Microwave breast cancer detection involves analysing the scattered waveforms of microwave signals that are propagated into the breast. We have developed a microwave-radar time-domain system and performed clinical trials using a prototype. This paper presents a classification architecture based on cost-sensitive support vector machines that is designed to process the signals measured by the 16-element multi-static antenna array. We examine the performance of the classifier by applying it to measurements performed on tissue-mimicking breast phantoms.

Random finite set (RFS) based filters such as the cardinalized probability hypothesis density (CPHD) filter have been successfully applied to the problem of single sensor multitarget tracking. Various multisensor extensions of these filters have been proposed in the literature, but exact update equations for the multisensor CPHD filter have not been identified. In this paper, we provide the update equations and propose an approximate implementation. The exact implementation of the multisensor CPHD filter is infeasible even for very simple scenarios. We develop an algorithm that greedily searches for the most likely groups of measurement subsets. This enables a computationally tractable implementation. Numerical simulations are performed to compare the proposed filter implementation with other random finite set based filters.

We introduce an online, time-dependent clustering algorithm that employs a dynamic probabilistic topic model. The proposed algorithm can handle data that evolves over time and strives to capture the evolution of clusters in the dataset. It addresses the case where the entire dataset is not available at once (e.g., the case of data streams) but an up-to-date clustering of the data at any given time is required. One of the main challenges of the data stream setting is that the computational cost and memory overhead must stay bounded as the number of data points increases. Our proposed algorithm has a Dirichlet process-based generative component combined with a sequential Monte Carlo sampler for posterior inference. We also introduce a novel modification to the sampling process, called targeted sampling, which enhances the performance of the SMC sampler. We test the performance of our algorithm with both synthetic and real datasets.

Even in the absence of a physical network infrastructure, it is important that mobile devices can take advantage of the opportunities presented to them to maintain communication. Opportunistic networks are a class of mobile networks that must rely on unscheduled sporadic meetings between nodes to achieve communication. The main challenge in these networks is to route messages towards a destination with high delivery probability, low average latency and efficient usage of network resources. Despite the numerous proposals for opportunistic routing protocols, there are relatively few analytical characterizations of their behaviour. In this paper, we propose both centralized and decentralized message forwarding algorithms that, under relatively strong assumptions about the network's behaviour, minimize the expected latency to a destination. After presenting proofs of the optimality of our proposed algorithms, we confirm their improvement in system performance through numerical simulation.

Radio-frequency (RF) tomography is the method of tracking targets using received signal-strength (RSS) measurements for RF transmissions between multiple sensor nodes. When the targets are near the line-of-sight path between two nodes, they are more likely to cause substantial attenuation or amplification of the RF signal. In this paper, we develop a measurement model for multitarget tracking using RF tomography in indoor environments and apply it successfully for tracking up to three targets. We compare several multitarget tracking algorithms and examine performance in the two scenarios when the number of targets is 1) known and constant, and 2) unknown and time varying. We demonstrate successful tracking for experimental data collected from sensor networks deployed in three different indoor environments posing different tracking challenges. For the fixed number of targets, the best algorithm achieves a root-mean-squared error tracking accuracy of approximately 0.3 m for a single target, 0.7 m for two targets and 0.8 m for three targets. Tracking using our proposed model is more accurate than tracking using previously proposed observation models; more importantly, the model does not require the same degree of training.

Radio Frequency (RF) tomographic tracking is the process of tracking moving targets by analyzing changes of attenuation in wireless transmissions. This paper presents a novel sequential Monte Carlo (SMC) method for RF tomographic tracking of a single target using a wireless sensor network. The algorithm incorporates on-line Expectation Maximization (EM) to estimate model parameters. Based on experimental measurements, we introduce a new measurement model for the attenuation caused by a target. We assess performance through numerical simulation and demonstrate that it significantly outperforms previous RF tomographic tracking procedures.

This paper introduces a distributed auxiliary particle filter for target tracking in sensor networks. Nodes maintain a shared particle filter by coming to a consensus about the likelihoods associated with each particle using the selective gossip procedure. Selective gossip provides a mechanism to efficiently identify the particles with largest weights and focus communication on sharing these important weights. We demonstrate through simulations that the algorithm performs well; compared to state-of-the-art approaches it either significantly improves the accuracy at the expense of a small increase in communication overhead, or achieves comparable accuracy with much lower communication overhead.

This paper analyzes the rate of convergence of greedy gossip with eavesdropping (GGE). In previous work, we proposed GGE, a fast gossip algorithm based on exploiting the broadcast nature of wireless communications rather than location information. Assuming all transmissions are wireless broadcasts, nodes can keep track of their neighbors' values by eavesdropping on their communications. Then, when it comes time to gossip, a node greedily and myopically gossips with the neighbor whose value is most different from its own, rather than with a randomly chosen neighbor. Previously, we have proved that GGE converges to the average consensus on connected network topologies and demonstrated that GGE outperforms standard randomized gossip (RG). In this paper we study the rate of convergence of GGE in terms of network voracity which is a topology-dependent constant analogous to the second-largest eigenvalue characterization for RG. Simulations demonstrate that the convergence rate of GGE is superior to existing average consensus algorithms such as geographic gossip.

Existing Internet traffic classification techniques achieve impressively low misclassification rates, but do not provide performance guarantees for particular classes of interest. In this paper, we propose two novel online traffic classifiers - one based on Neyman-Pearson classification and one based on the Learning Satisfiability (LSAT) framework - that can provide class-specific performance guarantees on the false alarm and false discovery rates, respectively. We also present a preprocessor for our classifiers that predicts, after the reception of only a small number of packets, whether a flow will be 'large' (as defined by a network operator). Only these resource-intensive flows are passed to the classifier, greatly reducing the computation burden imposed. We validate our methodology by testing our approaches using traffic data provided by an ISP.

The average consensus problem in the distributed signal processing context is addressed by linear iterative algorithms, with asymptotic convergence to the consensus. The convergence of the average consensus for an arbitrary weight matrix satisfying the convergence conditions is unfortunately slow restricting the use of the developed algorithms in applications. In this paper, we propose the use of linear extrapolation methods in order to accelerate distributed linear iterations. We provide analytical and simulation results that demonstrate the validity and effectiveness of the proposed scheme. Finally, we report simulation results showing that the generalized version of our algorithm, when a grid search for the unknown optimum value of mixing parameter is used, significantly outperforms the optimum consensus algorithm based on weight matrix optimization.

This paper investigates a class of learning problems called learning satisfiability (LSAT) problems, where the goal is to learn a set in the input (feature) space that satisfies a number of desired output (label/response) constraints. LSAT problems naturally arise in many applications in which one is interested in the class of inputs that produce desirable outputs, rather than simply a single optimum. A distinctive aspect of LSAT problems is that the output behavior is assessed only on the solution set, whereas in most statistical learning problems output behavior is evaluated over the entire input space. We present a novel support vector machine (SVM) algorithm for solving LSAT problems and apply it to a synthetic data set to illustrate the impact of the LSAT formulation.

Video-on-Demand (VoD) services are very user-friendly, but also complex and resource demanding. Deployments involve careful design of many mechanisms where content attributes and usage models should be taken into account. We define, and propose a methodology to solve, the VoD Equipment Allocation Problem of determining the number and type of streaming servers with directly attached storage (VoD servers) to install at each potential location in a metropolitan area network topology such that deployment costs are minimized. We develop a cost model for VoD deployments based on streaming, storage and transport costs and train a parametric function that maps the amount of available storage to a worst-case hit ratio. We observe the impact of having to determine the amount of storage and streaming cojointly, and determine the minimum demand required to deploy replicas as well as the average hit ratio at each location. We observe that common video-on-demand server configurations lead to the installation of excessive storage, because a relatively high hit-ratio can be achieved with small amounts of storage so streaming requirements dominate.

We address the problem of efficient end-to-end network monitoring of path metrics in communication networks. Our goal is to minimize the number of measurements or monitors required to maintain an acceptable estimation accuracy. We present a framework based on diffusion wavelets and nonlinear estimation. Our procedure involves the development of a diffusion wavelet basis that is adapted to the monitoring problem. This basis exploits spatial and temporal correlations in the measured phenomena to provide a compressible representation of the path metrics. The framework employs nonlinear estimation techniques using l1 minimization to generate estimates for the unmeasured paths. We describe heuristic approaches for the selection of the paths that should be monitored, or equivalently, where hardware monitors should be located. We demonstrate how our estimation framework can improve the efficiency of end-to-end delay estimation in IP networks and reduce the number of hardware monitors required to track bit-error rates in all-optical networks (networks with no electrical regenerators).

High-speed backbones are regularly affected by various kinds of network anomalies, ranging from malicious attacks to harmless large data transfers. Different types of anomalies affect the network in different ways, and it is difficult to know a priori how a potential anomaly will exhibit itself in traffic statistics. In this paper we describe an online, sequential, anomaly detection algorithm, that is suitable for use with multivariate data. The proposed algorithm is based on the kernel version of the recursive least squares algorithm. It assumes no model for network traffic or anomalies, and constructs and adapts a dictionary of features that approximately spans the subspace of normal behaviour. The algorithm raises an alarm immediately upon encountering a deviation from the norm. Through comparison with existing block-based offline methods based upon Principal Component Analysis, we demonstrate that our online algorithm is equally effective but has much faster time-to-detection and lower computational complexity. We also explore minimum volume set approaches in identifying the region of normality.

This paper introduces a feedback control system based on Smith's principle for bandwidth reservation and scheduling of all-photonic networks with large propagation delay. The approach is also applicable to any single-hop network with significant signalling delay (such as satellite-TDMA systems). Scheduling in wide-area networks must be based on predictions of traffic demand and the resultant errors can lead to instability and unfairness. The feedback control system we propose reduces the effect of prediction errors, increases the speed of the response to the sudden changes in traffic arrival rates, and improves fairness through equalization of queue-lengths.

Internal switches in all-photonic networks do not perform data conversion into the electronic domain, thereby eliminating a potential capacity bottleneck, but the inability to perform efficient optical buffering introduces network scheduling challenges. In this paper we focus on the problem of scheduling fixed-length frames in all-photonic star-topology networks with the goal of minimizing rejected demand. We formulate the task as an optimization problem and characterize its complexity. We describe the minimum rejection algorithm (MRA), which minimizes total rejection, and demonstrate that the fair matching algorithm (FMA) minimizes the maximum percentage rejection of any connection. We analyze through OPNET simulation the rejection and delay performance.

Abstract — The popularity of peer-to-peer (P2P) networks makes them an attractive target to the creators of viruses and other malicious code. Recently a number of viruses designed specifically to spread via P2P networks have emerged. Pollution has also become increasingly prevalent as copyright holders inject multiple decoy versions in order to impede item distribution. In this paper we derive deterministic epidemiological models for the propagation of a P2P virus through a P2P network and the dissemination of pollution. We report on discrete simulations that provide some verification that the models remain sufficiently accurate despite variations in individual peer conduct to provide insight into the behaviour of the system. The paper examines the steady-state behaviour and illustrates how the models may be used to estimate in a computationally efficient manner how effective object reputation schemes will be in mitigating the impact of viruses and preventing the spread of pollution. I.

Wireless sensor-actuator networks (SANETs), in which nodes perform actions (actuation) in response to sensor measurements and shared information, have great potential in medical and agricultural applications. In this paper, we focus on the problem of using distributed sensed data to design actuation strategies in order to elicit a desired response from the environment, whilst attempting to minimize the communication in the network. Our methodology is based on batch Q-learning; we describe a distributed approach for learning dyadic regression trees to estimate the Q-functions from collected data. Analysis and simulation indicate that substantial communication savings that can be achieved through distributed learning without significant performance deterioration. The simulations also reveal that the performance of our technique depends strongly on the amount of training data available

Video-on-demand (VoD) service providers are intensely interested in transport, storage, streaming and caching in content delivery networks. Today's 5,000-hour library may grow toward the 750,000-hour "long tail" movie and TV-series catalog. We propose a method to calculate how much of a library should be cached. Much previous work focused on theoretical caching concepts, or the dynamics of cache filling and reclamation. Our method explicitly considers the impact of the available video server equipment; we present a VoD design tool comprising a novel cost function, hit ratio estimation and heuristic

Information about the topology and link-level characteristics of a network is critical for many applications including network diagnostics and management. However, this information is not always directly accessible; subnetworks may not cooperate in releasing information and widespread local measurement can be prohibitively expensive. Network tomographic techniques obviate the need for network cooperation, but the majority assume probing from a single source, which imposes scalability limitations because sampling traffic is concentrated on network links close to the source. We describe a multiple source, end-to-end sampling architecture that uses coordinated transmission of carefully engineered multipacket probes to jointly infer logical topology and estimate link-level performance characteristics. We commence by demonstrating that the general multiple source, multiple destination tomography problem can be formally reduced to the two source, two destination case, allowing the immediate generalization of any sampling techniques developed for the simpler, smaller scenario. We then describe a method for testing whether links are shared in the topologies perceived by individual sources, and describe how to fuse the measurements in the shared case to generate more accurate estimates of the link-level performance statistics

This paper addresses the evaluation of causal relationships in large-scale wireless sensor/actuator networks (WSAN). WSAN consist of nodes that can monitor the environment (sensors), actively interact with the environment (actuators), or perform both functions. Such networks, together with the monitored environment, form complex, large-scale dynamic systems. It can be important to identify the causal relationships between variables in these systems; this provides individual actuator nodes with information about the probable consequences of local action on the global monitored environment. Such knowledge is critical to the formation of coordinated action plans, as might be required, for example, in distributed treatment strategies in medical or agricultural applications.

Sensor networks have exciting potential applications in agriculture and medicine, where after the application of treatment, it is beneficial not merely to track the response but to assess the causal impact of the treatment reception. We describe a distributed algorithm for the evaluation of the average causal effect of treatment reception upon response. Our procedure applies the expectation-maximization algorithm across a graphical model of the system, using local message-passing techniques. The key collaborative step in the algorithm is simple message aggregation and averaging, which we perform over a tree network topology. Finally, for completeness purposes, we describe a simple framework for the construction and maintenance of the tree topology that provides a robust mechanism for executing the algorithm using spread-spectrum or ultra-wideband communication.

The problem of identifying topology and inferring link-level performance parameters such as packet drop rate or delay variance using only end-to-end measurements is commonly referred to as network tomography. This paper describes a collaborative framework for performing network tomography on topologies with multiple sources and multiple destinations, without assuming the topology to be known. Using multiple sources potentially provides a more accurate and refined characterization of the internal network. We present a novel multiple source active measurement procedure using a semirandomized probing scheme and packet arrival order measurements which do not require precise synchronization between the participating hosts. A decision-theoretic framework is developed enabling the joint characterization of topology and internal performance. We design a statistical test based on the generalized likelihood ratio test and Wilks' theorem. The test quantifies the tradeoff between network topology complexity and performance estimation, and identifies when measurements made by the two sources can be combined to achieve reduced variance performance estimates. The performance and efficacy of the algorithm are assessed through ns-2 simulations and experiments over the Internet

Several recent price-based congestion control schemes require relatively accurate path price estimates for successful operation. The proposed addition of the two-bit explicit congestion notification (CCN) field in the IP header provides routers with a mechanism for conveying price information. Recently, two proposals have emerged for probabilistic packet marking at the routers; the proposals allow receivers to estimate path price from the fraction of marked packets. In this paper we introduce an alternative deterministic marking scheme for encoding path price. Under our approach, each router quantizes the price of its outgoing link to a fixed number of bits. We then make use of the IP identification (IPid) field to map data packets to different probe types, and each probe type calculates a partial sum of the path price bits. A router deduces its marking behaviour according to the IPid and the TTL (time to live) field of each packet. We evaluate the performance of our algorithm in terms of its error in representing the end-to-end price, and compare it to probabilistic marking. We show that based on empirical Internet traffic characteristics, our algorithm performs better when estimating path price using small blocks of packets. We also derive the probability distribution of the error for our scheme, and provide a relatively simple bound on its maximum mean-squared error.

This paper describes two methodologies for performing distributed particle filtering in a sensor network. It considers the scenario in which a set of sensor nodes make multiple, noisy measurements of an underlying, time-varying state that describes the monitored system. The goal of

Knowledge of network topology is useful for understanding the structure of the Internet, for developing and testing new protocols, and as prior information to network tomography algorithms. Building on existing techniques for inferring a single-source tree topology using end-to-end measurements, we address the problem of merging multiple tree topologies. We develop a multiple source active probing methodology and statistical framework for testing whether the paths from two sources to two receivers branch at a common internal node. This information can then be used to determine where portions of the tree topology from one source to a set of receivers overlap with the tree topology from a different source to the same set of receivers. The algorithm uses a novel random probing structure and easily made measurements of packet arrival order. As a result, we do not require precise time synchronization among the participating hosts. Successful experiments performed over a university LAN and over the Internet verify that our methodology is versatile and robust.

The substantial overhead of performing global Internet monitoring motivates techniques for inferring spatially localized information about performance using only host-based, end-to-end measurements. In this paper, we present a novel methodology for inferring queuing delay distributions across internal links in the network based solely on unicast, end-to-end measurements. A key feature of our new approach is that it is nonparametric, meaning that no a priori limit is placed on the number of unknown parameters used to model the delay distributions. The nonparametric approach is required in order to accurately estimate the wide variety of internal delay distributions. The methodology is formulated according to a recently proposed nonparametric, wavelet-based density estimation method in combination with an expectation-maximization optimization algorithm that employs a novel fast Fourier transform implementation. We perform network level ns simulations to verify the accuracy of the estimation procedure.

Network tomography is a process for inferring "internal" link-level delay and loss performance information based on end-to-end (edge) network measurements. These methods require knowledge of the network topology; therefore a first crucial step in the tomography process is topology identification. This paper considers the problem of discovering network topology solely from host-based, unicast measurements, without internal network cooperation. First, we introduce a novel delay-based measurement scheme that does not require clock synchronization, making it more practical than other previous proposals. In contrast to methods that rely on network cooperation , our methodology has the potential to identify layer two elements (provided they are logical topology branching points and induce some measurable delay). Second, we propose a maximum penalized likelihood criterion for topology identification. This is a global optimality criterion, in contrast to other recent proposals for topology identification that employ suboptimal, pair-merging strategies. We develop a novel Markov Chain Monte Carlo (MCMC) procedure for rapid determination of the most likely topologies. The performance of our new probing scheme and identification algorithm is explored through simulation and Internet experiments.

On-line, spatially localized information about internal network performance can greatly assist dynamic routing algorithms and traffic transmission protocols. However, it is impractical to measure network traffic at all points in the network. A promising alternative is to measure only at the edge of the network and infer internal behavior from these measurements. We concentrate on the estimation and localization of internal delays based on end-to-end delay measurements from sources to receivers. We develop an EM algorithm for computing MLE of the internal delay distributions in cases where the network dynamics are stationary over the observation period. For time-varying cases, we propose a sequential Monte Carlo procedure capable of tracking non-stationary delay characteristics. Simulations are included to demonstrate the promise of these techniques.

The paper presents a new method for characterizing communication network performance based solely on passive traffic monitoring at the network edge. More specifically, we devise a novel expectation-maximization (EM) algorithm to infer internal packet loss rates (at routers inside the network) using only observed end-to-end (source to receiver) loss rates. The major contributions of this paper are three-fold: we formulate a passive monitoring procedure for network loss inference based on end-to-end packet pair observations, we develop a statistical modeling and computation framework for inferring internal network loss characteristics, and we evaluate the performance with realistic network simulations.

We present an efficient method for robustly calculating time-frequency attributes of a signal, including instantaneous mean frequency, bandwidth, and kurtosis. Most current approaches involve a costly intermediate step of computing a (highly oversampled) 2-D bilinear time-frequency representation (TFR), which is then collapsed to the 1-D attribute. Using the principles of hybrid linear/bilinear time-frequency analysis, we propose computing attributes as nonlinear combinations of the (barely oversampled) linear Gabor transform of the signal. The method is both computationally efficient and accurate-it performs as well as the best bilinear techniques based on adaptive TFRs. To illustrate, we calculate an attribute of a seismic cross-section.

We address the problem of signal decomposition. We specify signal components by the property that their energies are localised and disjoint in the time-frequency plane. Rather than modelling the signal directly, we represent the time-frequency energy of the signal using a finite mixture model. This model is used to develop a partitioning of the time-frequency plane, allowing the application of time-frequency filtering to isolate components. Modelling energy rather than specifying a dictionary of allowable waveforms imposes fewer constraints on what a component may be. We demonstrate how the approach can be applied in the context of vibration analysis, where we wish to isolate the structure of individual bending waves travelling through a beam.

Poor directional selectivity, a major disadvantage of the 2D separable discrete wavelet transform (DWT), has heretofore been circumvented either by using highly redundant, nonseparable wavelet transforms or by using restrictive designs to obtain a pair of wavelet trees. In this paper, we demonstrate that superior directional selectivity may be obtained with no redundancy in any separable wavelet transform. We achieve this by projecting the wavelet transform coefficients onto the Softy space of signals and decimating before processing. A novel reconstruction step guarantees perfect reconstruction within this critically-sampled framework.

Ideally, kernels used to generate bilinear time-frequency distributions (TFD) should be signal-dependent, and optimised independently at every location in the time-frequency (TF) plane. This poses an extremely severe computational burden. A compromise is proposed in this paper: time-varying kernels are optimised for specific regions in the time-frequency plane. The regions, designed to isolate separate components comprising the signal, are determined by modelling the TFD using a finite mixture model of Gaussian distributions. The parameters of the model are estimated using a combination of the expectation-maximisation algorithm and functional merging. The regional optimisation provides improved separation and resolution of closely-spaced components when compared to methods using a solely time-varying kernel, without incurring an overwhelming computational expense.