In recent years, there have been exciting and accelerated developments in AI with novel developments in foundation models, deep learning, new AI applications across numerous verticals, and more. In addition, the pace of adoption of these innovations driven by both academic and industry research labs has sped up with both big tech companies and startups looking to deliver value-differentiated products and services. With Generative AI (GenAI) garnering significant attention, the second edition of the I2S workshop focuses on two aspects: First, bringing together AI thought leaders from academia, big tech, and startups to discuss the opportunities, use-case themes, challenges, and risks of GenAI in various business verticals; and Second, bringing together startup founders to share experiences and lessons learned in commercializing GenAI innovations into successful enterprises highlighting challenges through the entire commercial journey - from productization to acquiring customers, building a team, and securing funding.

Batch Normalization (BN) is a popular technique for training Deep Neural Networks (DNNs). BN uses scaling and shifting to normalize activations of mini-batches to accelerate convergence and improve generalization. The recently proposed Iterative Normalization (IterNorm) method improves these properties by whitening the activations iteratively using Newton’s method. However, since Newton’s method initializes the whitening matrix independently at each training step, no information is shared between consecutive steps. In this work, instead of exact computation of whitening matrix at each time step, we estimate it gradually during training in an online fashion, using our proposed Stochastic Whitening Batch Normalization (SWBN) algorithm. We show that while SWBN improves the convergence rate and generalization of DNNs, its computational overhead is less than that of IterNorm. Due to the high efficiency of the proposed method, it can be easily employed in most DNN architectures with a large number of layers. We provide comprehensive experiments and comparisons between BN, IterNorm, and SWBN layers to demonstrate the effectiveness of the proposed technique in conventional (many-shot) image classification and few-shot classification tasks.

The predominant paradigm for using machine learning models on a device is to train a model in the cloud and perform inference using the trained model on the device. However, with increasing numbers of smart devices and improved hardware, there is interest in performing model training on the device. Given this surge in interest, a comprehensive survey of the field from a device-agnostic perspective sets the stage for both understanding the state of the art and for identifying open challenges and future avenues of research. However, on-device learning is an expansive field with connections to a large number of related topics in AI and machine learning (including online learning, model adaptation, one/few-shot learning, etc.). Hence, covering such a large number of topics in a single survey is impractical. This survey finds a middle ground by reformulating the problem of on-device learning as resource constrained learning where the resources are compute and memory. This reformulation allows tools, techniques, and algorithms from a wide variety of research areas to be compared equitably. In addition to summarizing the state of the art, the survey also identifies a number of challenges and next steps for both the algorithmic and theoretical aspects of on-device learning.

Optimizing deep neural networks using gradient-based techniques such as SGD and Adam have some well documented problems including the high volatility, low stability and non-monotonicity of performance improvements during training. One approach that addresses these problems in convex settings is Polyak-Ruppert Averaging (PRA) which averages the model weights distributions over gradient updates. But this technique (and its variants) do not scale for non-convex settings and their use for large neural networks involves a tradeoff between increased stability and optimal convergence. In this paper, we introduce techniques centered around periodic sampling of model weights that improve upon gradient update methods (minibatch SGD, Momentum, Adam) for a variety of vision problems (classification, detection, segmentation). Compared to existing PRA approaches, the proposed algorithms provide better, faster and more robust convergence and training performance with only a slight increase in computation time. Importantly, these techniques are independent of the neural network model, gradient optimization methods and existing optimal training policies, and converge in a less volatile fashion with performance improvements that are approximately monotonic. We conduct a variety of experiments to quantify these improvements and identify scenarios where these techniques could be more useful.

This paper proposes an adaptive metric selection strategy called diagonal Barzilai-Borwein (DBB) stepsize for the popular Variable Metric Proximal Gradient (VM-PG) algorithm [1], [2]. The proposed approach better captures the local geometry of the problem while keeping the per-step computation cost similar to the widely used scalar Barzilai-Borwein (BB) stepsize. We provide the theoretical convergence analysis for VM-PG using DBB stepsize. Finally, our empirical results show ~10 - 40 % improvement in convergence times for the VM-PG using DBB compared to the BB stepsize for different machine learning problems on several datasets.

With recent advances in learning algorithms and hardware development, autonomous cars have shown promise when operating in structured environments under good driving conditions. However, for complex, cluttered, and unseen environments with high uncertainty, autonomous driving systems still frequently demonstrate erroneous or unexpected behaviors that could lead to catastrophic outcomes. Autonomous vehicles should ideally adapt to driving conditions; while this can be achieved through multiple routes, it would be beneficial as a first step to be able to characterize driveability in some quantified form. To this end, this paper aims to create a framework for investigating different factors that can impact driveability. Also, one of the main mechanisms to adapt autonomous driving systems to any driving condition is to be able to learn and generalize from representative scenarios. The machine learning algorithms that currently do so learn predominantly in a supervised manner and consequently need sufficient data for robust and efficient learning. Therefore, we also perform a comparative overview of 54 public driving datasets that enable learning and publish this dataset index at https://sites.google.com/view/driveability-survey-datasets. Specifically, we categorize the datasets according to the use cases and highlight the datasets that capture the complicated and hazardous driving conditions, which can be better used for training robust driving models. Furthermore, by discussions of what driving scenarios are not covered by the existing public datasets and what driveability factors need more investigation and data acquisition, this paper aims to encourage both targeted dataset collection and the proposal of novel driveability metrics that enhance the robustness of autonomous cars in adverse environments.

Tuning machine learning models at scale, especially finding the right hyperparameter values, can be difficult and time-consuming. In addition to the computational effort required, this process also requires some ancillary efforts including engineering tasks (e.g., job scheduling) as well as more mundane tasks (e.g., keeping track of the various parameters and associated results). We present Auptimizer, a general Hyperparameter Optimization (HPO) framework to help data scientists speed up model tuning and bookkeeping. With Auptimizer, users can use all available computing resources in distributed settings for model training. The user-friendly system design simplifies creating, controlling, and tracking of a typical machine learning project. The design also allows researchers to integrate new HPO algorithms. To demonstrate its flexibility, we show how Auptimizer integrates a few major HPO techniques (from random search to neural architecture search). The code is available at https://github.com/LGE-ARC-AdvancedAI/auptimizer.

Mobile edge computing allows resource-constrained mobile devices to offload their computationally-intensive tasks to powerful devices at the edge of the network. Designing an optimal offloading strategy for deciding whether to execute a task locally or at an edge device has attracted much attention recently. The existing literature focuses mostly on settings with a single mobile device or a single edge device, making the results unsuitable for real-world situations. This paper considers a setting consisting of multiple non-cooperative mobile devices and multiple edge devices, and aims to design offloading policies that minimize the task drop rate as well as the execution delay without requiring information about the dynamics of the environment such as channel models and task arrival rates at mobile devices. This non-cooperative resource allocation problem is only partially observable to each mobile device. We propose a method to mitigate the partial observability and then apply a deep reinforcement learning-based policy that progressively learns the dynamics of the environment as well as the long-term consequences of decisions. Numerical results demonstrate that the proposed algorithm significantly reduces the task drop rate compared to existing offloading policies, while minimizing energy and computation cost of each mobile device.

We introduce the notion of learning from contradictions, a.k.a Universum learning, for multiclass problems and propose a novel formulation for multiclass universum SVM (MU-SVM). We show that learning from contradictions (using MU-SVM) incurs lower sample complexity compared to multiclass SVM (M-SVM) by deriving the Natarajan dimension for sample complexity for PAC-learnability of MU-SVM. We also propose an analytic span bound for MU-SVM and demonstrate its utility for model selection resulting in $\sim 2-4 \times$ faster computation times than standard resampling techniques. We empirically demonstrate the efficacy of MU- SVM on several real world datasets achieving $>$ 20\% improvement in test accuracies compared to M-SVM. Insights into the underlying behavior of MU-SVM using a histograms-of-projections method are also provided.

NoSQL data-stores are commonly used to provide flexibility and availability for big data handling. However, there is a lack of comprehensive studies about which NoSQL data-store performs the best from the two scalability aspects, (scale-up, and scale-out), in a distributed and parallel processing environment. This paper compares the popular NoSQL data-stores (Cassandra, HBase, and MongoDB) and analyzes the resulting performance. Our experiments measure throughput, latency, and run-time of the evaluated data-stores on a big data set that consist of standard benchmarking workloads. Our results provide that the performance of each NoSQL data-store varies according to two main factors, (a) the type of executed operation, (read, scan, update, write, and insert), and (b) the level of distribution.

In this paper, we consider the problem of event prediction with multi-variate time series data consisting of heterogeneous (continuous and categorical) variables. The complex dependencies between the variables combined with asynchronicity and sparsity of the data makes the event prediction problem particularly challenging. Most state-of-art approaches address this either by designing hand-engineered features or breaking up the problem over homogeneous variates. In this work, we formulate the (rare) event prediction task as a classification problem with a novel asymmetric loss function and propose an end-to-end deep learning algorithm over symbolic representations of time-series. Symbolic representations are fed into an embedding layer and a Long Short Term Memory Neural Network (LSTM) layer which are trained to learn discriminative features. We also propose a simple sequence chopping technique to speed-up the training of LSTM for long temporal sequences. Experiments on real-world industrial datasets demonstrate the effectiveness of the proposed approach.

Large-scale analysis of data, especially in the industrial domain, is frequently a predictive exercise with a focus on extracting underlying correlations in the data. However, such analyses are often susceptible to biases and confounding effects. This paper investigates the use of causal data mining techniques where in the aforementioned biases can be mitigated. We investigate the application of such inference techniques to estimate the cause and effect relationships in an industrial setting particularly in the case when the outcome class is rare. Further, we discuss the underlying causal structure assumptions associated with each technique. Finally, we apply these techniques on a real world dataset obtained from an industrial setting.

Statistical models are commonly fit to bulk datasets, and they are applied in quasi real-time to previously unseen data. Challenges lie not only in fitting these models to data, but also in keeping track of their development and deployment process. It is common practice to re-engineer data pre-processing functions that were created during model development in order to build a version for deployment that works on streams of data. This approach is error-prone and inefficient. In this paper, we present our Model Deployment and Execution Framework (MDEF), to tackle these challenges in response to the volume, velocity, and variety of big data.

Most machine learning algorithms involve solving a convex optimization problem. Traditional in-memory convex optimization solvers do not scale well with the increase in data. This paper identifies a generic convex problem for most machine learning algorithms and solves it using the Alternating Direction Method of Multipliers (ADMM). Finally such an ADMM problem transforms to an iterative system of linear equations, which can be easily solved at scale in a distributed fashion. We implement this framework in Apache Spark and compare it with the widely used Machine Learning LIBrary (MLLIB) in Apache Spark 1.3.

One of the objectives of designing feature selection learning algorithms is to obtain classifiers that depend on a small number of attributes and have verifiable future performance guarantees. There are few, if any, approaches that successfully address the two goals simultaneously. To the best of our knowledge, such algorithms that give theoretical bounds on the future performance have not been proposed so far in the context of the classification of gene expression data. In this work, we investigate the premise of learning a conjunction (or disjunction) of decision stumps in Occam's Razor, Sample Compression, and PAC-Bayes learning settings for identifying a small subset of attributes that can be used to perform reliable classification tasks. We apply the proposed approaches for gene identification from DNA microarray data and compare our results to those of the well-known successful approaches proposed for the task. We show that our algorithm not only finds hypotheses with a much smaller number of genes while giving competitive classification accuracy but also having tight risk guarantees on future performance, unlike other approaches. The proposed approaches are general and extensible in terms of both designing novel algorithms and application to other domains.

Gadolinium-enhancing lesions in brain magnetic resonance imaging of multiple sclerosis (MS) patients are of great interest since they are markers of disease activity. Identification of gadolinium-enhancing lesions is particularly challenging because the vast majority of enhancing voxels are associated with normal structures, particularly blood vessels. Furthermore, these lesions are typically small and in close proximity to vessels. In this paper, we present an automatic, probabilistic framework for segmentation of gadolinium-enhancing lesions in MS using conditional random fields. Our approach, through the integration of different components, encodes different information such as correspondence between the intensities and tissue labels, patterns in the labels, or patterns in the intensities. The proposed algorithm is evaluated on 80 multimodal clinical datasets acquired from relapsing-remitting MS patients in the context of multicenter clinical trials. The experimental results exhibit a sensitivity of 98% with a low false positive lesion count. The performance of the proposed algorithm is also compared to a logistic regression classifier, a support vector machine and a Markov random field approach. The results demonstrate superior performance of the proposed algorithm at successfully detecting all of the gadolinium-enhancing lesions while maintaining a low false positive lesion count.

The analysis of execution traces can be useful in many software engineering activities including debugging, feature enhancement, performance analysis, and any other task that requires some degree of understanding of the way a system behaves. Traces, however, tend to be considerably large, which often hinders effective analysis of their content. There is a need to investigate ways to help software engineers find and understand important information conveyed in a trace despite the trace being massive. Motivated by the work done in the area of text mining, we propose, in this paper, a trace exploration approach based on examining the trace execution phases. The approach consists of automatically identifying relevant information about the phases as well as the ability to provide an efficient representation of the flow of phases by detecting redundant phases using a cosine similarity metric. We applied our approach to large traces generated from two different systems and were able to quickly understand their content and extract higher level views that characterize the essence of the information conveyed in these traces.

We propose a general theoretical framework for analyzing differentially expressed genes and behavior patterns from two homogenous short time-course data. The framework generalizes the recently proposed Hilbert-Schmidt Independence Criterion (HSIC)-based framework adapting it to the time-series scenario by utilizing tensor analysis for data transformation. The proposed framework is effective in yielding criteria that can identify both the differentially expressed genes and time-course patterns of interest between two time-series experiments without requiring to explicitly cluster the data. The results, obtained by applying the proposed framework with a linear kernel formulation, on various data sets are found to be both biologically meaningful and consistent with published studies.

We derive risk bounds for the randomized classifiers in Sample Compressions settings where the classifier-specification utilizes two sources of information viz. the compression set and the message string. By extending the recently proposed Occam’s Hammer principle to the data-dependent settings, we derive point-wise versions of the bounds on the stochastic sample compressed classifiers and also recover the corresponding classical PAC-Bayes bound. We further show how these compare favorably to the existing results.

We propose a formulation of the Decision Tree learning algorithm in the Compression settings and derive tight generalization error bounds. In particular, we propose Sample Compression and Occam's Razor bounds. We show how such bounds, unlike the VC dimension or Rademacher complexities based bounds, are more general and can also perform a margin-sparsity trade-off to obtain better classifers. Potentially, these risk bounds can also guide the model selection process and replace traditional pruning strategies.

We design a new learning algorithm for the Set Covering Ma- chine from a PAC-Bayes perspective and propose a PAC-Bayes risk bound which is minimized for classifiers achieving a non trivial margin-sparsity trade-off.

We propose a “soft greedy” learning algorithm for building small conjunctions of simple threshold functions, called rays, defined on single real-valued attributes. We also propose a PAC-Bayes risk bound which is minimized for classifiers achieving a non-trivial tradeoff between sparsity (the number of rays used) and the mag- nitude of the separating margin of each ray. Finally, we test the soft greedy algorithm on four DNA micro-array data sets.