We introduce a comprehensive benchmark to assess the analogical reasoning capabilities of large language models (LLMs) on complex analogy tasks that go beyond conventional formats with single correct answers. Unlike standard benchmarks that assume a singular ground truth, our framework presents a four-way multiple-choice analogy task in which all target options are semantically plausible. Leveraging concept pairs from Wikidata and AnalogyKB, we construct analogy instances enriched with multiple overlapping relational structures, where the relations are mined with RAG and ranked in salience through a GPT-4-assisted Max-Diff survey. To enable systematic evaluation, we propose three complementary semantic measures i.e. ranked relational overlap, context embedding similarity, and prototypicality; each grounded in established literature on analogical reasoning. Our experiments span a range of LLMs, evaluated under zero-shot, few-shot, and knowledge-enhanced prompting conditions. While models such as GPT-4 perform well on embedding-based and prototypicality-based measures, they consistently underperform when tasked with capturing fine-grained relational mappings. These results reveal that, despite their impressive surface-level semantic fluency, current LLMs exhibit notable limitations in structured relational reasoning.

We address the real problem of safe, robust, adaptive resource oversubscription in uncertain environments with our proposed novel technique of chance-constrained imitation learning. Our objective is to enhance resource efficiency while ensuring safety against congestion risk. Traditional supervised or forecasting models are ineffective in learning adaptive oversubscription policies, and conventional online optimization or reinforcement learning is difficult to deploy on real systems. Offline policy learning methods, such as Imitation Learning (IL) can leverage historical resource utilization telemetry data to learn effective policies if we can ensure robustness and safety from the underlying uncertainty in the domain, and thus the data. Our work investigates the nature of this uncertainty, how it can be quantified and proposes a novel chance-constrained IL that implicitly models such uncertainty in a principled manner via additional knowledge in the form of stochastic constraints on the associated risk, to learn provably safe and robust policies. We show empirically a substantial improvement (~ 3-4×) in capacity efficiency and congestion safety in test as well as real deployments.

Operating server components beyond their voltage and power design limit (i.e., overclocking) enables improving performance and lowering cost for cloud workloads. However, overclocking can significantly degrade component lifetime, increase power draw, and cause power capping events, eventually diminishing the performance benefits. In this paper, we characterize the impact of overclocking on cloud workloads by studying their profiles from production deployments. Based on the characterization insights, we propose SmartOClock, the first distributed overclocking management platform specifically designed for cloud environments. SmartOClock is a workload-aware scheme that relies on power predictions to heterogeneously distribute the power budgets across its servers based on their needs and then enforce budget compliance locally, per-server, in a decentralized manner. SmartOClock reduces the tail latency by 9%, application cost by 30% and total energy consumption by 10% for latencysensitive microservices on a 36-server deployment. Simulation analysis using production traces show that SmartOClock reduces the number of power capping events by up to 95% while increasing the overclocking success rate by up to 62%. We also describe lessons from building a first-of-its-kind overclockable cluster in Microsoft Azure for production experiments.

The rise of social media has amplified the visibility and impact of hate speech, prompting the development of NLP solutions to identify both explicit and implicit forms of hate speech.These approaches assess toxicity in isolation, neglecting context and limiting models to sentence-level understanding.Therefore we study, how contextual factors influence perceived toxicity, thereby anchoring assessments in a more nuanced semantic framework.We introduce a novel synthetic data generation pipeline designed to create contextutterance pairs at scale with controlled polarity.This pipeline can enhance existing hate speech datasets by adding contextual information to utterances, either preserving or altering their polarity, and also generate completely new pairs from seed statements.We utilised both features to create Toximatics, a dataset that includes context-dependent utterances and it's toxicity score.To address biases in state-of-the-art hate datasets, which often skew towards specific sensitive topics such as politics, race, and gender, we propose a method to generate neutral utterances typical of various social settings.These are then contextualized to show how neutrality can shift to toxicity or benignity depending on the surrounding context.Toximatics' approach to hate speech detection extends beyond the sentence level, rendering it suitable for discourse analysis and also revealing that current models underperform on this dataset.

We investigate the problem of smart operational efficiency, at scale, in Machine Learning models for Big Data streams, in context of embedded AI applications, by learning optimal early exits. Embedded AI applications that employ deep neural models depend on efficient model inference at scale, especially on resource-constrained hardware. Recent vision/text/audio models are computationally complex with huge parameter spaces and input samples typically pass through multiple layers, each with large tensor computations, to produce valid outputs. Generally, in most real scenarios, AI applications deal with big data streams, such as streams of audio signals, static images and/or high resolution video frames. Deep ML models powering such applications have to continuously perform inference on such big data streams for varied tasks such as noise suppression, face detection, gait estimation and so on. Ensuring efficiency is challenging, even with model compression techniques since they reduce model size but often fail to achieve scalable inference efficiency over continuous streams. Early exits enable adaptive inference by extracting valid outputs from any pre-final layer of a deep model which significantly boosts efficiency at scale since many of the input instances need not be processed at all the layers of a deep model, especially for big streams. Suitable early exit structure design (number + positions) is a difficult but crucial aspect in improving efficiency without any loss in predictive performance, especially in context of big streams. Naive manual early exit design that does not consider the hardware capacity or data stream characteristics is counterproductive. We propose NASEREX framework that leverages Neural architecture Search (NAS) with a novel saliency-constrained search space and exit decision metric to learn suitable early exit structure to augment Deep Neural models for scalable efficient inference on big image streams. Optimized exit-augmented models perform $\approx 2.5 \times$ faster having $\approx 4 \times$ aggregated lower effective FLOPs, with no significant accuracy loss.

Biological samples are routinely analyzed for microbe concentration. The samples are diluted, loaded onto established host cell cultures, and incubated. If infectious agents are present in the samples, they form circular spots that do not contain the host cells. Each spot is assumed to be originated from a single microbial unit such as a bacterial colony forming unit or viral plaque forming unit. The undiluted sample concentration is estimated by counting the spots and back-calculating. Counting the number of spots by trained technicians is currently the gold standard but it is laborious, subjective, and hard to scale. This paper presents a new automated algorithm for spot counting, Localized and Sequential Thresholding (LoST). Validation studies showed that LoST performance was comparable with manual counting and outperformed several existing tools on images with overlapping spots. The LoST algorithm employs sequential thresholding through a two-stage segmentation and borrows information across all images from the same dilution series to fine-tune the count and identify right censoring. The algorithm increases the efficiency of the spot counting and the quality of the downstream analysis, especially when coupled with an appropriate statistical serial dilution model to enhance the undiluted sample concentration estimation procedure.

Natural language generation (NLG) models can propagate social bias towards particular demography. Though several studies investigated bias from data and model, NLG task distinctively uses stochastic decoder that can positively or negatively impact the bias-sensitive tokens initially predicted by the model. To address this gap in research, we present an extensive analysis of bias from decoding techniques for open-domain language generation considering the entire decoding space. We analyze to what extent bias metrics like toxicity and sentiment are impacted by the individual components of decoder algorithms. To this extent, we also analyze the trade-off between bias scores and human-annotated generation quality throughout the decoder space. Together, these methods reveal the imperative of testing inference time bias and provide evidence on the usefulness of inspecting the entire decoding spectrum.

Designing suitable deep model architectures for AI-driven on-device apps and features, at par with evolving mobile hardware and complex target scenarios, is difficult. Though Neural Architecture Search (NAS/AutoML) has made this easier by automated architecture learning from data saving substantial manual effort, yet it has major limitations, in context of mobile devices, including model-hardware alignment, prohibitive search times and divergence from primary target objective(s). So, we propose AUTOCOMET that can learn the most suitable DNN architecture optimized for varied types of device hardware and task contexts, ≈ 3× faster. Our novel co-regulated shaping reinforcement controller together with the high fidelity hardware meta-behavior predictor produces a smart, fast NAS framework that adapts to context via a generalized formalism for any kind of multi-criteria optimization.

The modern era of artificial intelligence is mostly driven by Deep Neural Network (DNN). As a result most of the intelligent/smart apps running on edge devices (mobile phones, televisions etc.) use DNN for their predictive ability. DNN based prediction suffers with operational overheads, which is the summation of model loading latency and inference latency. Model loading latency affects the first response of the DNN powered apps, whereas inference latency affects the subsequent responses. As apps switching has become a common practice among edge device users, so it is of utmost interest to make the switching smooth by reducing the model loading latency. In this paper asymmetrical DNN modularization is proposed as a potential solution. The proposed method solves two distinct problems (a). Improves the model loading latency by parallel loading of all the modules (child models) of given DNN model. (b). Helps in on-device training by keeping the live gradients only for last child model. The decision about modularization index and their corresponding positions are taken by reinforcement learning unit (RLU). RLU takes into account the available hardware resources on-device (eg. h/w threads), loading latency of each layer on dedicated compute units. In response, it provides the best modularization index k and their corresponding positions$\vec{p}$specific to the DNN model and device, where$P=(p_{1},p_{2},\ \ldots,p_{k})$and$p_{i}$is the end position of child$i$. The proposed method has shown significant loading time improvement (up to 7X) on popular DNNs, used for camera use-cases. Along with improving the loading latency the proposed modularization method facilitates for On-device personalization by separating the module with trainable layers and loading them particularly while training on-device.

While Generative Adversarial Networks (GANs) have accelerated the use of generative modelling within the machine learning community, most of the adaptations of GANs are restricted to images. The use of GANs to generate clinical data has been rare due to the inability of GANs to faithfully capture the intrinsic relationships between features given a small amount of observational data. We hypothesize and verify that this challenge can be mitigated by incorporating rich domain knowledge in the form of expert advice in the generative process. Specifically, we propose human-allied GANs that uses correlation advice from humans to create synthetic clinical data. We construct a system that takes a symbolic representation of the expert advice and converts it into constraints on correlation of the features during the generative process. Our empirical evaluation demonstrates (a) the superiority of our approach over other GAN models, (b) the importance of incorporating advice over instance noise and (c) an initial framework for incorporation of privacy in our model while capturing the relationships between features.

Counting the number of true instances of a clause is arguably a major bottleneck in relational probabilistic inference and learning. We approximate counts in two steps: (1) transform the fully grounded relational model to a large hypergraph, and partially-instantiated clauses to hypergraph motifs; (2) since the expected counts of the motifs are provably the clause counts, approximate them using summary statistics (in/outdegrees, edge counts, etc). Our experimental results demonstrate the efficiency of these approximations, which can be applied to many complex statistical relational models, and can be significantly faster than state-of-the-art, both for inference and learning, without sacrificing effectiveness.

One of the key advantages of Inductive Logic Programming systems is the ability of the domain experts to provide background knowledge as modes that allow for efficient search through the space of hypotheses. However, there is an inherent assumption that this expert should also be an ILP expert to provide effective modes. We relax this assumption by designing a graphical user interface that allows the domain expert to interact with the system using Entity Relationship diagrams. These interactions are used to construct modes for the learning system. We evaluate our algorithm on a probabilistic logic learning system where we demonstrate that the user is able to construct effective background knowledge on par with the expert-encoded knowledge on five data sets.

Over the past decade, exploiting relations and symmetries within probabilistic models has been proven to be surprisingly effective at solving large scale data mining problems. One of the key operations inside these lifted approaches is counting - be it for parameter/structure learning or for efficient inference. Typically, however, they just count exploiting the logical structure using adhoc operators. This paper investigates whether ‘Compilation to Graph Databases’ could be a practical technique for scaling lifted probabilistic inference and learning methods. We demonstrate that the proposed approach achieves reasonable speed-ups for both inference and learning, without sacrificing performance.