This research explores integration of a Large Language Model (LLM) fine-tuned to conversationally control the user interface (UI) for a Semantic Automation Layer (SAL). We condense SAL capabilities from prior work and prioritize with business analysts and data engineers via a Kano model, before implementing a prototypical UI. We augment the UI with our conversational engine and propose In-situ Prompt Engineering and learn from Human Feedback to smoothen the interaction and manipulation of UI through natural language commands. To evaluate the efficacy and usability of conversational control in various use-case scenarios, we conduct and report on an empirical interaction design user study. Our findings provide evidence supporting enhanced user engagement and satisfaction. We also observe significant increase of trust in AI after working with our conversational UI. This work generates areas for further refinement and research towards more intelligent, highly-integrated conversational UIs even beyond our application within Semantic Automation. We discuss our findings and point out next steps paving the way for future research and development in creating more intuitive and adaptive user interfaces.

In recent years, the automation of machine learning and data science (AutoML) has attracted significant attention. One under-explored dimension of AutoML is being able to automatically utilize domain knowledge (such as semantic concepts and relationships) located in historical code or literature from the problem's domain. In this paper, we demonstrate Semantic Feature Discovery, which enables users to interactively explore features semantically discovered from existing data science code and external knowledge. It does so by detecting semantic concepts for a given dataset, and then using these concepts to determine relevant feature engineering operations from historical code and knowledge.

Pretrained models from self-supervision are prevalently used in fine-tuning downstream tasks faster or for better accuracy. However, gaining robustness from pretraining is left unexplored. We introduce adversarial training into self-supervision, to provide general-purpose robust pretrained models for the first time. We find these robust pretrained models can benefit the subsequent fine-tuning in two ways: i) boosting final model robustness; ii) saving the computation cost, if proceeding towards adversarial fine-tuning. We conduct extensive experiments to demonstrate that the proposed framework achieves large performance margins (eg, 3.83% on robust accuracy and 1.3% on standard accuracy, on the CIFAR-10 dataset), compared with the conventional end-to-end adversarial training baseline. Moreover, we find that different self-supervised pretrained models have diverse adversarial vulnerability. It inspires us to ensemble several pretraining tasks, which boosts robustness more. Our ensemble strategy contributes to a further improvement of 3.59% on robust accuracy, while maintaining a slightly higher standard accuracy on CIFAR-10. Our codes are available at https://github.com/TAMU-VITA/Adv-SS-Pretraining.

Learning to optimize (L2O) is gaining increased attention because classical optimizers require laborious, problem-specific design and hyperparameter tuning. However, there are significant performance and practicality gaps between manually designed optimizers and existing L2O models. Specifically, learned optimizers are applicable to only a limited class of problems, often exhibit instability, and generalize poorly. As research efforts focus on increasingly sophisticated L2O models, we argue for an orthogonal, under-explored theme: improved training techniques for L2O models. We first present a progressive, curriculum-based training scheme, which gradually increases the optimizer unroll length to mitigate the well-known L2O dilemma of truncation bias (shorter unrolling) versus gradient explosion (longer unrolling). Secondly, we present an off-policy imitation learning based approach to guide the L2O learning, by learning from the behavior of analytical optimizers. We evaluate our improved training techniques with a variety of state-of-the-art L2O models and immediately boost their performance, without making any change to their model structures. We demonstrate that, using our improved training techniques, one of the earliest and simplest L2O models can be trained to outperform even the latest and most complex L2O models on a number of tasks. Our results demonstrate a greater potential of L2O yet to be unleashed, and prompt a reconsideration of recent L2O model progress. Our codes are publicly available at: https://github.com/VITA-Group/L2O-Training-Techniques.

As application demands for zeroth-order (gradient-free) optimization accelerate, the need for variance reduced and faster converging approaches is also intensifying. This paper addresses these challenges by presenting: a) a comprehensive theoretical analysis of variance reduced zeroth-order (ZO) optimization, b) a novel variance reduced ZO algorithm, called ZO-SVRG, and c) an experimental evaluation of our approach in the context of two compelling applications, black-box chemical material classification and generation of adversarial examples from black-box deep neural network models. Our theoretical analysis uncovers an essential difficulty in the analysis of ZO-SVRG: the unbiased assumption on gradient estimates no longer holds. We prove that compared to its first-order counterpart, ZO-SVRG with a two-point random gradient estimator could suffer an additional error of order $O(1/b)$, where $b$ is the mini-batch size. To mitigate this error, we propose two accelerated versions of ZO-SVRG utilizing variance reduced gradient estimators, which achieve the best rate known for ZO stochastic optimization (in terms of iterations). Our extensive experimental results show that our approaches outperform other state-of-the-art ZO algorithms, and strike a balance between the convergence rate and the function query complexity.

Multimedia stream mining applications require the identification of several different attributes in data content, and hence rely on a set of cascaded statistical classifiers to filter and process the data dynamically. In this paper, we introduce a novel methodology for configuring such cascaded classifier topologies, specifically binary classifier trees, in resource-constrained, distributed stream mining systems. Instead of traditional load shedding, our approach configures classifiers with optimized operating points after jointly considering the misclassification cost of each end-to-end class of interest in the tree, the resource constraints for every classifier, and the confidence level of each data object that is classified. The proposed approach allows for both intelligent load shedding as well as data replication based on available resources dynamically. We evaluate the algorithm on a sports video concept detection application and identify huge cost savings over load shedding alone. Additionally, we propose several distributed algorithms that enable each classifier in the tree to reconfigure itself based on local information exchange. We analyze the associated tradeoffs between convergence time, information overhead, and the cost efficiency of results achieved by each classifier for each of these algorithms.

Stream mining applications require the identification of several different attributes in data content and hence rely on a distributed set of cascaded statistical classifiers to filter and process the data dynamically. In this letter, we introduce a novel methodology for configuring cascaded classifier topologies, specifically binary classifier trees, with optimized operating points after jointly considering the misclassification cost of each end-to-end class of interest in the tree, the resource constraints for every classifier, and the confidence level of each data object that is classified. By configuring multiple operating points per classifier, we enable not only intelligent load shedding when resources are scarce but also intelligent replication of low confidence data across multiple edges when excess resources are available. Using a classifier tree constructed from support vector machine-based sports image classifiers, we verify huge cost savings and discuss how different classifier placements and costs can influence the gains obtained by various algorithms.

We design algorithms for optimally configuring, in terms of model complexity as well as operating point, a cascade of exclusive classifiers on distributed systems, given underlying resource constraints. Under rate-dependent constraints we design a Viterbi-like strategy to determine the optimal solution with significantly lower computational complexity than an exhaustive search. We illustrate the performance of this strategy on a cascade of classifiers for a speaker verification task, and highlight performance gains over using a single classifier.

Distributed stream processing systems offer a highly scalable and dynamically configurable platform for time-critical applications ranging from real-time, exploratory data mining to high performance transaction processing. Resource management for distributed stream processing systems is complicated by a number of factors processing elements are constrained by their producer-consumer relationships, data and processing rates can be highly bursty, and traditional measures of effectiveness, such as utilization, can be misleading. In this paper, we propose a novel distributed, adaptive control algorithm that maximizes weighted throughput while ensuring stable operation in the face of highly bursty workloads. Our algorithm is designed to meet the challenges of extreme-scale stream processing systems, where overprovisioning is not an option, by making the best use of resources even when the proffered load is greater than available resources. We have implemented our algorithm in a real-world distributed stream processing system and a simulation environment. Our results show that our algorithm is not only self-stabilizing and robust to errors, but also outperforms traditional approaches over a broad range of buffer sizes, processing graphs, and burstiness types and levels.

Networks of classifiers are capturing the attention of system and algorithmic researchers because they offer improved accuracy over single model classifiers, can be distributed over a network of servers for improved scalability, and can be adapted to available system resources. This work provides a principled approach for the optimized allocation of system resources across a networked chain of classifiers. We begin with an illustrative example of how complex classification tasks can be decomposed into a network of binary classifiers. We formally define a global performance metric by recursively collapsing the chain of classifiers into one combined classifier. The performance metric trades off the end-to-end probabilities of detection and false alarm, both of which depend on the resources allocated to each individual classifier. We formulate the optimization problem and present optimal resource allocation results for both simulated and state-of-the-art classifier chains operating on telephony data.

Systems designed to extract time-critical information from large volumes of unstructured data must include the ability, both from an architectural and algorithmic point of view, to filter out unimportant data that might otherwise overwhelm the available resources. This paper presents an approach for data filtering to reduce computation in the context of a distributed speech processing architecture designed to detect or identify speakers. Here, filtering means either dropping and ignoring data or passing it on for further processing. The goal of the paper is to show that when the filter is designed to select and pass on a subset of the input data that best preserves the ability to recognize a specific desired speaker, or group of speakers, a large percentage of the data can be ignored while being able to preserve most of the accuracy

Stream processing applications have recently gained significant attention in the networking and database community. At the core of these applications is a stream processing engine that performs resource allocation and management to support continuous tracking of queries over collections of physically-distributed and rapidly-updating data streams. While numerous stream processing systems exist, there has been little work on understanding the performance characteristics of these applications in a distributed setup. In this paper, we examine the performance bottlenecks of streaming data applications, in particular the Linear Road stream data management benchmark, in achieving good performance in large-scale distributed environments, using the Stream Processing Core (SPC), a stream processing middleware we have developed. First, we present the design and implementation of the Linear Road benchmark on the SPC middleware. SPC has been designed to scale to tens of thousands of processing nodes, while supporting concurrent applications and multiple simultaneous queries. Second, we identify the main performance bottlenecks in the Linear Road application in achieving scalability and low query response latency. Our results show that data locality, buffer capacity, physical allocation of processing elements to infrastructure nodes, and packaging for transporting streamed data are important factors in achieving good application performance. Though we evaluate our system primarily for the Linear Road application, we believe it also provides useful insights into the overall system behavior for supporting other distributed and large-scale continuous streaming data applications. Finally, we examine how SPC can be used and tuned to enable a very efficient implementation of the Linear Road application in a distributed environment.

In this paper, we propose a semantic routing and filtering framework for large-scale monitoring of video streams. Our goal is to build a distributed system that at any given time is capable of simultaneously monitoring the content of multiple video streams being transmitted over the Internet (or a proprietary network). A key design requirement of such a system is the ability to handle tens of gigabytes of multimedia data per second. Traditional techniques have an important limitation. Once a bottleneck in terms of CPU or storage is reached, data is dropped indiscriminately.. In this paper, we propose distributed real-time semantic filters to route and filter video data. We propose a mechanism to alter the accuracy of classification with the complexity of execution; thus avoiding system failure during periods of overload. We propose a set of novel video features that perform better than our previous semantic classifiers. This system is capable of classifying over a hundred concepts. Experiments on 190 hours of pre-stored and live video streams validate the effectiveness of the proposed system.

Resource management mechanisms for large-scale, globally distributed network services need to assign groups of clients to servers according to network location and expected load generated by these clients. Current proposals address network location and traffic modeling separately. We develop a novel clustering technique that addresses both network proximity and traffic modeling. Our approach combines techniques from network-aware clustering, location inference, and spatial analysis. We conduct a large, measurement-based study to identify and evaluate Web traffic clusters. Our study links millions of Web transactions collected from two world-wide sporting event Websites, with millions of network delay measurements to thousands of Internet address clusters. Because our techniques are equally applicable to other traffic types, they are useful in a variety of wide-area distributed computing optimizations, and Internet modeling and simulation scenarios.

Peering allows service providers to handle traffic surges without over-provisioning, reduce the cost of dedicated infrastructure, and leverage the specialization and prices of partner providers. We develop a peering system for multi-provider content delivery based on a cost-optimized peer selection algorithm. We formulate a cost model for evaluating competing peering strategies, and use measurement data collected from globally distributed network probe stations, large-scale Web sites, and existing service provider infrastructures to empirically evaluate proposed peering strategies. Our analysis shows that our peer selection algorithm is significantly more efficient than greedy alternatives, in terms of minimizing service cost and respecting network delay and server capacity thresholds, over a broad range of real-world scenarios.

Service providers are increasingly turning to federated approaches to meet customer demand while limiting the risk of large-scale computing infrastructure investments. In such models, service providers could meet the demands of peak processing periods or specialized application environments by offloading to a partner provider. While a number of distributed systems initiatives, including Grid computing, rely on such cooperatives, models for effective capacity planning and settlement in such environments has largely been ignored. In this paper, we develop a model for settlement and capacity planning in a multi-provider computing cooperative. We introduce two pricing tariffs - linear and burstable pricing - that are commonly used in the industry, and compare their relative merits.

In this demonstration we present the ARMS system which enables secure and adaptive rich media streaming to a large-scale, heterogeneous client population. The ARMS system dynamically adapts streams to available bandwidth, client capabilities, packet loss, and administratively imposed policies - all while maintaining full content security. The ARMS system is completely standards compliant and to our knowledge is the first such end-to-end MPEG-4-based system.

Internet server selection mechanisms attempt to optimize, subject to a variety of constraints, the distribution of client requests to a geographically and topologically diverse pool of servers. Research on server selection has thus far focused primarily on techniques for choosing a server from a group administered by single entity, like a content distribution network provider. In a federated, multi-provider computing system, however, selection must occur over distributed server sets deployed by the participating providers, without the benefit of the full information available in the single-provider case. Intelligent server set selection algorithms will require a model of the expected performance clients would receive from a candidate server set.In this paper, we study whether the complex policies and dynamics of intelligent server selection can be effectively modeled in order to predict client performance for server sets. We introduce a novel server set distance metric, and use it in a measurement study of several million server selection transactions to develop simple models of existing server selection schemes. We then evaluate these models in terms of their ability to accurately predict performance for a second, larger set of distributed clients. We show that our models are able to predict performance within 20ms for over 90% of the observed samples. Our analysis demonstrates that although existing deployments use a variety of complex and dynamic server selection criteria, most of which are proprietary, these schemes can be modeled with surprising accuracy.

While there has been extensive research into wide-area load balancing for Web servers, dynamic server selection issues for rich media have largely been ignored We argue that streaming media imposes new challenges that are not met by current proposals for collecting and disseminating server and network path metrics. We define a model for quantitatively evaluating the network overhead of competing Internet measurement strategies, propose a novel strategy to better meet requirements for rich media, and use topologies representative of the Internet to show our strategy reduces total and average per link probe overhead by as much as 98%. We discuss additional advantages to our approach, including the ability to represent multiple, potentially streaming media specific, metrics; the ability to effect policy-based selection; and protection against denial of service attacks.

Interactive media applications have moved the ability to manipulate content as it is being streamed from a desirable feature to a required attribute for video servers.However, present day video server architectures continue to focus primarily on disk and network strategies for maximizing the number of concurrent streams while adhering to Quality of Service (QoS) requirements, If video servers are to manage large numbers of disparate streams exploiting a wide variety of processing algorithms, server architectures must be extended to support the integration of arbitrary isochronous processing algorithms.We believe server support for these stream processing graphs present new research challenges in video server buffer management, distributed stream control and QoS management.In this paper, we propose distributed stream control techniques which form a foundation for developing technology to address these new challenges.We have implemented the recommended techniques in the context of a commercial video server.