While vision-language models (VLMs) have demonstrated remarkable performance across various tasks combining textual and visual information, they continue to struggle with fine-grained visual perception tasks that require detailed pixel-level analysis. Effectively eliciting comprehensive reasoning from VLMs on such intricate visual elements remains an open challenge. In this paper, we present VipAct, an agent framework that enhances VLMs by integrating multi-agent collaboration and vision expert models, enabling more precise visual understanding and comprehensive reasoning. VipAct consists of an orchestrator agent, which manages task requirement analysis, planning, and coordination, along with specialized agents that handle specific tasks such as image captioning and vision expert models that provide high-precision perceptual information. This multi-agent approach allows VLMs to better perform fine-grained visual perception tasks by synergizing planning, reasoning, and tool use. We evaluate VipAct on benchmarks featuring a diverse set of visual perception tasks, with experimental results demonstrating significant performance improvements over state-of-the-art baselines across all tasks. Furthermore, comprehensive ablation studies reveal the critical role of multi-agent collaboration in eliciting more detailed System-2 reasoning and highlight the importance of image input for task planning. Additionally, our error analysis identifies patterns of VLMs' inherent limitations in visual perception, providing insights into potential future improvements. VipAct offers a flexible and extensible framework, paving the way for more advanced visual perception systems across various real-world applications.

Machine learning is evolving towards high-order models that necessitate pre-training on extensive datasets, a process associated with significant overheads. Traditional models, despite having pre-trained weights, are becoming obsolete due to architectural differences that obstruct the effective transfer and initialization of these weights. To address these challenges, we introduce a novel framework, QuadraNet V2, which leverages quadratic neural networks to create efficient and sustainable high-order learning models. Our method initializes the primary term of the quadratic neuron using a standard neural network, while the quadratic term is employed to adaptively enhance the learning of data non-linearity or shifts. This integration of pre-trained primary terms with quadratic terms, which possess advanced modeling capabilities, significantly augments the information characterization capacity of the high-order network. By utilizing existing pre-trained weights, QuadraNet V2 reduces the required GPU hours for training by 90% to 98.4% compared to training from scratch, demonstrating both efficiency and effectiveness.

Federated learning (FL) nowadays involves heterogeneous compound learning tasks as cognitive applications’ complexity increases. For example, a self-driving system hosts multiple tasks simultaneously (e.g., detection, classification, segmentation, etc.) and expects FL to retain life-long intelligence involvement. However, our analysis demonstrates that, when deploying compound FL models for multiple training tasks on a GPU, certain issues arise: As different tasks’ skewed data distributions and corresponding models cause highly imbalanced learning workloads, current GPU scheduling methods lack effective resource allocations; Therefore, existing FL schemes, only focusing on heterogeneous data distribution but runtime computing, cannot practically achieve optimally synchronized federation. To address these issues, we propose a full-stack FL optimization scheme to tackle both intra-device GPU scheduling and inter-device FL coordination for multi-task training. Specifically, our works illustrate two key insights in this research domain: Competitive resource sharing is beneficial for parallel model executions, and the proposed concept of “virtual resource” could effectively characterize and guide the practical per-task resource utilization and allocation; Additionally, architectural-level coordination improves FL performance by aligning task workloads with GPU utilization. Our experiments demonstrate that the FL performance could be significantly escalated. Specifically, we observed a 2.16×–2.38× increase in intra-device GPU training throughput and a 2.53×–2.80× boost in inter-device FL coordination efficiency across diverse multi-task scenarios.

Large language models (LLMs) have shown remarkable progress in understanding and generating natural language across various applications. However, they often struggle with resolving ambiguities in real-world, enterprise-level interactions, where context and domain-specific knowledge play a crucial role. In this demonstration, we introduce ECLAIR (Enhanced CLArification for Interactive Responses), a multi-agent framework for interactive disambiguation. ECLAIR enhances ambiguous user query clarification through an interactive process where custom agents are defined, ambiguity reasoning is conducted by the agents, clarification questions are generated, and user feedback is leveraged to refine the final response. When tested on real-world customer data, ECLAIR demonstrates significant improvements in clarification question generation compared to standard few-shot methods.

We present ECLAIR (Enhanced CLArification for Interactive Responses), a novel unified and end-to-end framework for interactive disambiguation in enterprise AI assistants. ECLAIR generates clarification questions for ambiguous user queries and resolves ambiguity based on the user's response. We introduce a generalized architecture capable of integrating ambiguity information from multiple downstream agents, enhancing context-awareness in resolving ambiguities and allowing enterprise specific definition of agents. We further define agents within our system that provide domain-specific grounding information. We conduct experiments comparing ECLAIR to few-shot prompting techniques and demonstrate ECLAIR's superior performance in clarification question generation and ambiguity resolution.

Transformers have emerged as the leading architecture in deep learning, proving to be versatile and highly effective across diverse domains beyond language and image processing. However, their impressive performance often incurs high computational costs due to their substantial model size. This paper focuses on compressing decoder-only transformer-based autoregressive models through structural weight pruning to improve the model efficiency while preserving performance for both language and image generation tasks. Specifically, we propose a training-free pruning method that calculates a numerical score with Newton's method for the Attention and MLP modules, respectively. Besides, we further propose another compensation algorithm to recover the pruned model for better performance. To verify the effectiveness of our method, we provide both theoretical support and extensive experiments. Our experiments show that our method achieves state-of-the-art performance with reduced memory usage and faster generation speeds on GPUs.

Yu Xia, Subhojyoti Mukherjee, Zhouhang Xie, Junda Wu, Xintong Li, Ryan Aponte, Hanjia Lyu, Joe Barrow, Hongjie Chen, Franck Dernoncourt, Branislav Kveton, Tong Yu, Ruiyi Zhang, Jiuxiang Gu, Nesreen K. Ahmed, Yu Wang, Xiang Chen, Hanieh Deilamsalehy, Sungchul Kim, Zhengmian Hu, Yue Zhao, Nedim Lipka, Seunghyun Yoon, Ting-Hao Kenneth Huang, Zichao Wang, Puneet Mathur, Soumyabrata Pal, Koyel Mukherjee, Zhehao Zhang, Namyong Park, Thien Huu Nguyen, Jiebo Luo, Ryan A. Rossi, Julian McAuley. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.

Chain-of-Thought (CoT) has been a widely adopted prompting method, eliciting impressive reasoning abilities of Large Language Models (LLMs). Inspired by the sequential thought structure of CoT, a number of Chain-of-X (CoX) methods have been developed to address challenges across diverse domains and tasks. In this paper, we provide a comprehensive survey of Chain-of-X methods for LLMs in different contexts. Specifically, we categorize them by taxonomies of nodes, i.e., the X in CoX, and application tasks. We also discuss the findings and implications of existing CoX methods, as well as potential future directions. Our survey aims to serve as a detailed and up-to-date resource for researchers seeking to apply the idea of CoT to broader scenarios.

Xintong Li, Junda Wu, Tong Yu, Rui Wang, Yu Wang, Xiang Chen, Jiuxiang Gu, Lina Yao, Julian McAuley, Jingbo Shang. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025.

Mehrab Tanjim, Yeonjun In, Xiang Chen, Victor Bursztyn, Ryan A. Rossi, Sungchul Kim, Guang-Jie Ren, Vaishnavi Muppala, Shun Jiang, Yongsung Kim, Chanyoung Park. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025.

Offline evaluation of LLMs is crucial in understanding their capacities, though current methods remain underexplored in existing research. In this work, we focus on the offline evaluation of the chain-of-thought capabilities and show how to optimize LLMs based on the proposed evaluation method. To enable offline feedback with rich knowledge and reasoning paths, we use knowledge graphs (KGs) (e.g., Wikidata5M) to provide feedback on the generated chain of thoughts. Due to the heterogeneity between LLM reasoning and KG structures, direct interaction and feedback from knowledge graphs on LLM behavior are challenging, as they require accurate entity linking and grounding of LLM-generated chains of thought in the KG. To address the above challenge, we propose an offline chain-of-thought evaluation framework, OCEAN, which models chain-of-thought reasoning in LLMs as a Markov Decision Process (MDP), and evaluate the policy’s alignment with KG preference modeling. To overcome the reasoning heterogeneity and grounding problems, we leverage on-policy KG exploration and reinforcement learning to model a KG policy that generates token-level likelihood distributions for LLM-generated chain-of-thought reasoning paths, simulating KG reasoning preference. Then we incorporate the knowledge-graph feedback on the validity and alignment of the generated reasoning paths into inverse propensity scores and propose KG-IPS estimator. Theoretically, we prove the unbiasedness of the proposed KG-IPS estimator and provide a lower bound on its variance. With the off-policy evaluated value function, we can directly enable off-policy optimization to further enhance chain-of-thought alignment. Our empirical study shows that OCEAN can be efficiently optimized for generating chain-of-thought reasoning paths with higher estimated values without affecting LLMs’ general abilities in downstream tasks or their internal knowledge.

As the size of Deep Neural Networks (DNNs) continues to grow, their runtime latency also scales. While model pruning and Neural Architecture Search (NAS) can effectively reduce the computation workload, their effectiveness fails to consistently translate into runtime latency reduction. In this paper, we identify the root cause behind the mismatch between workload reduction and latency reduction is GPU tail effect – a classic system issue caused by resource under-utilization in the last processing wave of the GPU. We conduct detailed DNN workload characterization and demonstrate the prevalence of GPU tail effect across different DNN architectures, and meanwhile reveal that the unique deep structure and the light-weight layer workload of DNNs exacerbate the tail effect for DNN inference. We then propose a tail-awareness design space enhancement and DNN optimization algorithm to optimize existing NAS and pruning designs and achieve better runtime latency and model accuracy performance. Extensive experiments show 11%-27% latency reduction over SOTA DNN pruning and NAS methods.

Junda Wu, Tong Yu, Xiang Chen, Haoliang Wang, Ryan Rossi, Sungchul Kim, Anup Rao, Julian McAuley. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.

Recent progress in computer vision-oriented neural network designs is mostly driven by capturing high-order neural interactions among inputs and features. And there emerged a variety of approaches to accomplish this, such as Transformers and its variants. However, these interactions generate a large amount of intermediate state and/or strong data dependency, leading to considerable memory consumption and computing cost, and therefore compromising the overall runtime performance. To address this challenge, we rethink the high-order interactive neural network design with a quadratic computing approach. Specifically, we propose QuadraNet — a comprehensive model design methodology from neuron reconstruction to structural block and eventually to the overall neural network implementation. Leveraging quadratic neurons’ intrinsic high-order advantages and dedicated computation optimization schemes, QuadraNet could effectively achieve optimal cognition and computation performance. Incorporating state-of-the-art hardware-aware neural architecture search and system integration techniques, QuadraNet could also be well generalized in different hardware constraint settings and deployment scenarios. The experiment shows that QuadraNet achieves up to 1.5 × throughput, 30% less memory footprint, and similar cognition performance, compared with the state-of-the-art high-order approaches.

Text-to-image generative models can generate high-quality humans, but realism is lost when generating hands. Common artifacts include irregular hand poses, shapes, incorrect numbers of fingers, and physically implausible finger orientations. To generate images with realistic hands, we propose a novel diffusion-based architecture called HanDiffuser that achieves realism by injecting hand embeddings in the generative process. HanDiffuser consists of two components: a Text-to-Hand-Params diffusion model to generate SMPL-Body and MANO-Hand parameters from input text prompts, and a Text-Guided Hand-Params-to-Image diffusion model to synthesize images by conditioning on the prompts and hand parameters generated by the previous component. We incorporate multiple aspects of hand representation, including 3D shapes and joint-level finger positions, orientations and articulations, for robust learning and reliable performance during inference. We conduct extensive quantitative and qualitative experiments and perform user studies to demonstrate the efficacy of our method in generating images with high-quality hands. Project page: https://supreethn.github.io/research/handiffuser/index.html

This paper studies the decentralized dynamic kernel learning problem where each agent in the network receives continuous streaming local data and works collaboratively to learn a non-linear function "on the fly" in a dynamic environment. We utilize the random feature (RF) mapping method to circumvent the curse of dimensionality issue in conventional kernel methods and reformulate the dynamic kernel learning problem as a dynamic parameter optimization problem, which is then efficiently solved by the Decentralized Dynamic Kernel Learning via ADMM (DDKL) framework. To further improve communication efficiency, we incorporate the quantization and censoring strategies in the communication stage and develop the Quantized and Communication-censored DDKL (QC-DDKL) algorithm. We theoretically prove that QC-DDKL can achieve the optimal sublinear regret $\mathcal{O}(\sqrt T )$ over T time slots. Simulation results also corroborate the learning effectiveness and the communication efficiency of the proposed method.

As deep learning continues to advance and is applied to increasingly complex scenarios, the demand for concurrent deployment of multiple neural network models has arisen. This demand, commonly referred to as multi-tenant computing, is becoming more and more important. However, even the most mature GPU-based computing systems struggle to adequately address the significant heterogeneity and complexity among concurrent models in terms of resource allocation and runtime scheduling. And this usually results in considerable resource utilization and throughput issues. To tackle these issues, this work proposes a set of optimization techniques that advance the granularity of computing management from both the spatial and temporal perspectives, specifically tailored to heterogeneous model compositions for deep learning inference and training. These techniques are further integrated as GACER --- an automated optimization framework that provides high-utilization, high-throughput, and low-latency multi-tenant computing support. And our experiments demonstrate that GACER significantly improves the overall resource utilization and consistently achieves outstanding speedups compared to native GPU computing frameworks and existing state-of-the-art optimization works.

Large language models (LLMs) have demonstrated powerful decision-making and planning capabilities in solving complicated real-world problems. LLM-based autonomous agents can interact with diverse tools (e.g., functional APIs) and generate solution plans that execute a series of API function calls in a step-by-step manner. The multitude of candidate API function calls significantly expands the action space, amplifying the critical need for efficient action space navigation. However, existing methods either struggle with unidirectional exploration in expansive action spaces, trapped into a locally optimal solution, or suffer from exhaustively traversing all potential actions, causing inefficient navigation. To address these issues, we propose ToolChain*, an efficient tree search-based planning algorithm for LLM-based agents. It formulates the entire action space as a decision tree, where each node represents a possible API function call involved in a solution plan. By incorporating the A$^*$ search algorithm with task-specific cost function design, it efficiently prunes high-cost branches that may involve incorrect actions, identifying the most low-cost valid path as the solution. Extensive experiments on multiple tool-use and reasoning tasks demonstrate that ToolChain* efficiently balances exploration and exploitation within an expansive action space. It outperforms state-of-the-art baselines on planning and reasoning tasks by 3.1% and 3.5% on average while requiring 7.35x and 2.31x less time, respectively.

Recent research works demonstrate that one of the significant factors for the model Out-of-Distirbution detection performance is the scale of the OOD feature representation field. Consequently, model ensemble emerges as a trending method to expand this feature representation field leveraging expected model diversity. However, by proposing novel qualitative and quantitative model ensemble evaluation methods (i.e., Loss Basin/Barrier Visualization and Self-Coupling Index), we reveal that the previous ensemble methods incorporate affine-transformable weights with limited variability and fail to provide desired feature representation diversity. Therefore, we escalate the traditional model ensemble dimensions (different weight initialization, data holdout, etc.) into distinct supervision tasks, which we name as Multi-Comprehension (MC) Ensemble. MC Ensemble leverages various training tasks to form different comprehensions of the data and labels, resulting in the extension of the feature representation field. In experiments, we demonstrate the superior performance of the MC Ensemble strategy in the OOD detection task compared to both the naive Deep Ensemble method and the standalone model of comparable size.

Tuple space search(TSS)-based packet classification is the keystone of network system. Previous studies accelerate TSS by partitioning tuples, combining trees and tuples, and merging tuples. However, they do not scale with the number of rules, resulting in a high memory footprint or update time. In this paper, we propose TupleRadar, a framework for accelerating TSS while ensuring low memory footprint and fast rule updates. Our key idea is to construct learned indexes for tuples, which inherently improve the lookup speed but ensure the advantages of TSS. Specifically, TupleRadar builds orderly hash table-based tuples and then constructs the updatable learned index. It provides a bounded memory footprint of the index structure as well. We have evaluated TupleRadar on multiple scales rule-sets. Experimental results show that TupleRadar outperforms previous solutions, reducing 46.66% lookup time and 61.53% memory footprint on average, by up to 86.70% and 88.95%. It also performs a competitive rule update speed.

The past neural network design has largely focused on feature \textit{representation space} dimension and its capacity scaling (e.g., width, depth), but overlooked the feature \textit{interaction space} scaling. Recent advancements have shown shifted focus towards element-wise multiplication to facilitate higher-dimensional feature interaction space for better information transformation. Despite this progress, multiplications predominantly capture low-order interactions, thus remaining confined to a finite-dimensional interaction space. To transcend this limitation, classic kernel methods emerge as a promising solution to engage features in an infinite-dimensional space. We introduce InfiNet, a model architecture that enables feature interaction within an infinite-dimensional space created by RBF kernel. Our experiments reveal that InfiNet achieves new state-of-the-art, owing to its capability to leverage infinite-dimensional interactions, significantly enhancing model performance.

This article focuses on online kernel learning over a decentralized network. Each agent in the network receives online streaming data and collaboratively learns a globally optimal nonlinear prediction function in the reproducing kernel Hilbert space (RKHS). To overcome the curse of dimensionality issue in traditional online kernel learning, we utilize random feature (RF) mapping to convert the nonparametric kernel learning problem into a fixed-length parametric one in the RF space. We then propose a novel learning framework, named online decentralized kernel learning via linearized ADMM (ODKLA), to efficiently solve the online decentralized kernel learning problem. To enhance communication efficiency, we introduce quantization and censoring strategies in the communication stage, resulting in the quantized and communication-censored ODKLA (QC-ODKLA) algorithm. We theoretically prove that both ODKLA and QC-ODKLA can achieve the optimal sublinear regret over time slots. Through numerical experiments, we evaluate the learning effectiveness, communication efficiency, and computation efficiency of the proposed methods.

Data-driven machine learning techniques have been advocated for signal detection in complex wireless environments. However, when applied to wideband spectrum sensing scenarios, they face practical challenges including very large data dimensionality, insufficient training data, and implicit inter-band dependencies. Current literature focuses on deep convolutional models, whose inherent model structure is not well suited for representing the diverse spectrum occupancy patterns of practical wideband networks, causing inefficient performance-complexity tradeoff and excessive sensing time. To address these issues, this paper develops a novel Spectrum Transformer with multi-task learning for wideband spectrum sensing at high sample efficiency. Empowered by the multi-head self-attention mechanism, the transformer architecture is designed to effectively learn both the inner-band spectral features and the inter-band spectrum occupancy correlations in the wideband regime. Simulations show that the proposed Spectrum Transformer outperforms the existing methods based on convolutional neural networks especially in the small-data case, by achieving higher sensing accuracy with an 89% reduction in model complexity.

There is increasing attention toward search indexes based on dense vector representations as they can encode latent semantic features and can be generated efficiently, thanks to the advances in representation learning. Building an index on vectors was addressed by methods like locality-sensitive hashing (LSH), Inverted File Index (IVF), and near-neighbor graphs and perfected for high performance. However, the current near-neighbor (NN) indexes cannot be directly used in a real recommendation engine where both learned dense features and the constraints-attributes are used. The existing methods are cascaded index with vector search followed by a naive matching or inverted index for filtering on attribute tokens. This filtering on post-vector search matched sets often limits the control on output size and adds latency due to repeated calls to the NN engine. We aim to make a single-stage retrieval model that can retrieve in a single pass and control the output size without compromising on latency. An NN index amenable to this should have a very similar structure to the attribute token-based inverted index. Hence, we develop an efficient constraint search engine based on the high dimensional sparse embeddings of semantic features added with attribute tokens. With this, we get an Inverted index-based Constraint Near Neighbor search-ICONN, where we retrieve search results with 100% match on query attributes and close to its semantic features. We achieve a better latency vs recall10@10 tradeoff compared to the standard NN search followed by attribute filtering.

Analyzing data and creating effective visualizations often requires extensive domain expertise. For users with less experience, it can be difficult to know how to get started with exploratory data analysis (EDA) and how to approach the code. Chatbots can reduce the gap between analysis outcomes and user expectations by leveraging multi-turn conversations to provide a more natural interface between the user and computer-agent. To inform the design of future visual EDA chatbots, we conduct a survey and interview study with ten potential users. Our results suggest that users want a visual EDA chatbot that can make exploratory data analysis easier, while also augmenting their knowledge of visualization and analysis techniques. Between the initial survey and post-interview questionnaire, we saw increased optimism overall for the usefulness and anticipated analytic ease of visual EDA chatbots. Based on these results, we identify four key design guidelines: future visual EDA chatbots should (1) understand the user’s data and intent, (2) respond with useful visualizations, (3) leverage the history of the visualizations and data, and (4) produce verifiable and shareable analysis processes.

Vanilla federated learning does not support learning in an online environment, learning a personalized model on each client, and learning in a decentralized setting. There are existing methods extending federated learning in each of the three aspects. However, some important applications on enterprise edge servers (e.g. online item recommendation at global scale) involve the three aspects at the same time. Therefore, we propose a new learning setting Decentralized Personalized Online Federated Learning that considers all the three aspects at the same time.In this new setting for learning, the first technical challenge is how to aggregate the shared model parameters from neighboring clients to obtain a personalized local model with good performance on each client. We propose to directly learn an aggregation by optimizing the performance of the local model with respect to the aggregation weights. This not only improves personalization of each local model but also helps the local model adapting to potential data shift by intelligently incorporating the right amount of information from its neighbors. The second challenge is how to select the neighbors for each client. We propose a peer selection method based on the learned aggregation weights enabling each client to select the most helpful neighbors and reduce communication cost at the same time. We verify the effectiveness and robustness of our proposed method on three real-world item recommendation datasets and one air quality prediction dataset.

Recommendation systems suggest items to users by predicting their preferences based on historical data. The industry traditionally handles large-scale recommendation requests by scaling the number of devices without much concern for a single device’s performance. However, there is a trend for recommendation systems to gradually move from a centralized service to an edge device. The edge-scale recommendation systems have distinct features that are different from traditional large-scale deployments, which poses different challenges to the acceleration of the recommendation system. In this paper, we focus on the edge-scale recommendation system and propose an inter-stage parallelism optimization method deployed on a single GPU. Experiments show that our framework could improve recommendation system throughput by 1.89×~2.2× for different datasets on the GPU.

In Internet Service Provider (ISP) networks, Amplified Reflection DDoS (AR-DDoS) attack is one of the main attack categories, which launches gigabytes of traffic with little effort and minimal cost. Thus, the mitigation of AR-DDoS attacks has been considered as a crucial part. In particular, such mitigation requires full coverage (i.e., mitigating AR-DDoS attacks launched from any location) and low overhead (i.e., mitigation should avoid high latency that degrades user experience). However, existing solutions suffer from either limited coverage or high overhead. In this paper, we propose Aigis, a distributed framework that offers full-coverage and low-overhead mitigation of AR-DDoS attacks. Our key idea is to co-design top-of-rack (ToR) switches and end-hosts, which offers line-rate packet processing performance and fine-grained view inherently, to jointly execute endpoint verification. Specifically, Aigis selectively offloads mitigation operations between ToR switches and end-hosts and implements a network-wide epoch synchronization mechanism to guarantee reliable verification. It efficiently coordinates ToR switches and end-hosts to execute the entire mitigation task. We have implemented Aigis on a testbed comprising 32×100 Gbps Tofino switches. Testbed experiments indicate that Aigis achieves complete full coverage and orders of magnitude lower host-side overhead compared to existing solutions.

Packet classification is a crucial component in computer networking. To achieve high throughput and low memory consumption, existing solutions apply different heuristics in each construction stage to build efficient decision trees. However, previous studies divide the tree construction process based on the scale of rule subsets which is indirect to the performance goal, leading to massive rule replication and high tree depth. In this paper, we propose MiCuts, a fine-grained framework for packet classification with both high speed and low memory footprint. Its key idea is directly utilizing rule replication and tree depth to divide the tree-building process into three stages, each with suitable optimization goals. First, it partitions rules and builds shallow semi-trees without rule replication via selecting effective bits. Second, it transforms the switching problem of heuristics into an ILP problem and aims to minimize memory consumption while ensuring high lookup speed. Third, it merges some nodes to eliminate memory explosion caused by splitting, where MiCuts combines splitting and linear search. Extensive experimental results on ClassBench show that MiCuts outperforms state-of-the-art approaches, improving lookup speed by 1.71× while reducing memory footprint by 74.4% on average.

Due to the often limited communication bandwidth of edge devices, most existing federated learning (FL) methods randomly select only a subset of devices to participate in training at each communication round. Compared with engaging all the available clients, such a random-selection mechanism could lead to significant performance degradation on non-IID (independent and identically distributed) data. In this paper, we present our key observation that the essential reason resulting in such performance degradation is the class-imbalance of the grouped data from randomly selected clients. Based on this observation, we design an efficient heterogeneity-aware client sampling mechanism, namely, Federated Class-balanced Sampling (Fed-CBS), which can effectively reduce class-imbalance of the grouped dataset from the intentionally selected clients. We first propose a measure of class-imbalance which can be derived in a privacy-preserving way. Based on this measure, we design a computation-efficient client sampling strategy such that the actively selected clients will generate a more class-balanced grouped dataset with theoretical guarantees. Experimental results show that Fed-CBS outperforms the status quo approaches in terms of test accuracy and the rate of convergence while achieving comparable or even better performance than the ideal setting where all the available clients participate in the FL training.

Learning from noisy labels is a challenge that arises in many real-world applications where training data can contain incorrect or corrupted labels. When fine-tuning language models with noisy labels, models can easily overfit the label noise, leading to decreased performance. Most existing methods for learning from noisy labels use static input features for denoising, but these methods are limited by the information they can provide on true label distributions and can result in biased or incorrect predictions. In this work, we propose the Dynamics-Enhanced Generative Model (DyGen), which uses dynamic patterns in the embedding space during the fine-tuning process of language models to improve noisy label predictions. DyGen uses the variational auto-encoding framework to infer the posterior distributions of true labels from noisy labels and training dynamics. Additionally, a co-regularization mechanism is used to minimize the impact of potentially noisy labels and priors. DyGen demonstrates an average accuracy improvement of 3.10% on two synthetic noise datasets and 1.48% on three real-world noise datasets compared to the previous state-of-the-art. Extensive experiments and analyses show the effectiveness of each component in DyGen. Our code is available for reproducibility on GitHub.

Convolutional Neural Networks (CNNs) are widely deployed in various embedded recognition applications. However, they demonstrate a considerable vulnerability to adversarial attacks, which leverage the well-designed perturbations to mislead the recognition results. Recently, for easier perturbation injection and higher attack effectiveness, the adversarial perturbations have been concentrated into a small area with various types and different data modalities. When defending such condensed adversarial attacks on the embedded recognition scenarios, most of the existing defense works highlight two critical issues. First, they are particularly designed for each individual condensed attack scenario, lacking enough versatility to accommodate attacks with different data modalities. Second, they rely on computation-intensive preprocessing techniques, which is impractical for time-sensitive embedded recognition scenarios. In this article, we propose LanCeX –a versatile and lightweight CNN defense solution against condensed adversarial attacks. By examining the CNN’s intrinsic vulnerability, we first identify the common attacking mechanism behind condensed adversarial attacks across different data modalities. Based on this mechanism, LanCeX can defend against various condensed attacks with the optimal computation workload in different recognition scenarios. Experiments show that LanCeX can achieve an average 91%, 85%, and 90% detection success rate and optimal adversarial mitigation performance in three recognition scenarios, respectively: image classification, object detection, and audio recognition. Moreover, LanCeX is at most 3× faster compared with the state-of-the-art defense methods, making it feasible to use with resource-constrained embedded systems.

Earlier recommendation techniques, such as Collaborative Filtering (CF), assume the users’ preferences do not change over time and strive to maximize the immediate reward. In recent studies, Reinforcement Learning (RL) has been used to make interactive recommendation systems that capture users’ preferences over time and maximize the long-term reward. However, these methods have two limitations. First, they assume that items are independently distributed and do not consider the relations between items. This assumption ignores the power of the relations between items for recommendation systems, as demonstrated by CF. RL-based methods rely primarily on users’ positive feedback to understand their preferences, and sampling is used to incorporate their negative feedback. In a practical setting, users’ negative feedback is just as crucial as their positive feedback for understanding their preferences. We present a novel Deep Reinforcement Learning (DRL) recommendation framework to address the limitations above. We specifically utilize the actor-critic paradigm, which considers the recommendation problem a sequential decision-making process to adapt to users’ behaviors and maximize the long-term reward. Motivated by the intuition that similar users like similar items, we extract the relations between items using CF and integrate it into our framework to boost overall performance. Instead of negative sampling, our proposed framework relies on all users’ positive and negative feedback to understand users’ preferences more accurately. Extensive experiments with our dataset and two public datasets demonstrate the effectiveness of our proposed framework.

We study the problem of extracting N -ary relation tuples from scientific articles.This task is challenging because the target knowledge tuples can reside in multiple parts and modalities of the document.Our proposed method RESEL decomposes this task into a two-stage procedure that first retrieves the most relevant paragraph/table and then selects the target entity from the retrieved component.For the highlevel retrieval stage, RESEL designs a simple and effective feature set, which captures multilevel lexical and semantic similarities between the query and components.For the low-level selection stage, RESEL designs a cross-modal entity correlation graph along with a multi-view architecture, which models both semantic and document-structural relations between entities.Our experiments on three scientific information extraction datasets show that RESEL outperforms state-of-the-art baselines significantly. 1

This paper proposes deep kernel learning networks with multiple learning paths (DKL-MLP) for nonlinear function approximation. Leveraging the random feature (RF) mapping technique, kernel methods can be implemented as a two-layer neural network, at drastically reduced workload on weight training. Motivated by the representation power of the deep architecture in deep neural networks, we devise a vanilla deep kernel learning network (DKL) by applying RF mapping at each layer and learn the last layer only. To improve the learning performance of DKL, we add multiple trainable paths to DKL and develop the DKL-MLP method so that some implicit information from earlier hidden layers to the output layer can be learned. We prove that both DKL and DKL-MLP permit universal representation of a wide variety of interesting functions with arbitrarily small error and have no bad local minimum. Numerical experiments on both regression and classification tasks are provided to demonstrate the learning performance and computational efficiency of the proposed methods.

Reconfigurable intelligent surfaces (RISs) are regarded as one of the most promising techniques in the sixth-generation (6G) mobile communication networks. With the feature of smartly tuning the electromagnetic environment, RISs provide a possibility for ubiquitous and high-precision localization in 6G. However, proper system models for large indoor RIS-assisted networks and high-precision localization algorithms are still missing. In this paper, we propose a RIS-assisted downlink millimeter-wave (mmWave) indoor localization framework based on segment-by-segment far-field assumption. In addition, a brand new coarse-to-fine localization algorithm with low-complexity grid design is provided. Numerical results show that millimeter-level localization precision is achieved under the RIS-assisted indoor scenarios, which reveals that RIS can provide a solid support for accurate localization in the 6G era.

Many works focus on the model’s static parameter optimization (e.g., filters and weights) for CNN inference acceleration. Compared to parameter sparsity, feature map sparsity is per-input related which has better adaptability. The practical sparsity patterns are non-structural and randomly located on feature maps with non-identical shapes. However, the existing feature map sparsity works take computing efficiency as the primary goal, thereby they can only remove structural sparsity and fail to match the above characteristics. In this paper, we develop a novel sparsity computing scheme called FalCon, which can well adapt to the practical sparsity patterns while still maintaining efficient computing. Specifically, we first propose a decomposed convolution design that enables a fine-grained computing unit for sparsity. Additionally, a decomposed convolution computing optimization paradigm is proposed to convert the sparse computing units to practical acceleration. Extensive experiments show that FalCon achieves at most 67.30% theoretical computation reduction with a neglected accuracy drop while accelerating CNN inference by 37%.

Current state-of-the-art object detectors can have significant performance drop when deployed in the wild due to domain gaps with training data. Unsupervised Domain Adaptation (UDA) is a promising approach to adapt detectors for new domains/environments without any expensive label cost. Previous mainstream UDA works for object detection usually focused on image-level and/or feature-level adaptation by using adversarial learning methods. In this work, we show that such adversarial-based methods can only reduce domain style gap, but cannot address the domain content gap that is also important for object detectors. To overcome this limitation, we propose the SC-UDA framework to concurrently reduce both gaps: We propose fine-grained domain style transfer to reduce the style gaps with finer image details preserved for detecting small objects; Then we leverage the pseudo label-based self-training to reduce content gaps; To address pseudo label error accumulation during self-training, novel optimizations are proposed, including uncertainty-based pseudo labeling and imbalanced mini-batch sampling strategy. Experiment results show that our approach consistently outperforms prior state-of-the-art methods (up to 8.6%, 2.7% and 2.5% mAP on three UDA benchmarks).

Personal daily mobility trajectories/traces like Google Location Service integrates many valuable information from individuals and could benefit a lot of application scenarios, such as pandemic control and precaution, product recommendation, customized user profile analysis, traffic management in smart cities, etc. However, utilizing such personal mobility data faces many challenges since users’ private information, such as home/work addresses, can be unintentionally leaked. In this work, we build an FL system for transportation mode prediction based on personal mobility data. Utilizing FL-based training scheme, all user’s data are kept in local without uploading to central nodes, providing high privacy preserving capability. At the same time, we could train accurate DNN models that is close to the centralized training performance. The resulted transportation mode prediction system serves as a prototype on user’s traffic mode classification, which could potentially benefit the transportation data analysis and help make wise decisions to manage public transportation resources.

Nowadays, the evolution of deep learning and cloud service significantly promotes neural network-based mobile applications. Although intelligent and prolific, those applications still lack certain flexibility: for classification tasks, neural networks are generally trained with vast classification targets to cover various utilization contexts. However, only partial classes are practically inferred due to individual mobile user preference and application specificity, which causes unnecessary computation consumption. Thus, we proposedCaptorX—a class-adaptive convolutional neural network (CNN) reconfiguration framework to adaptively prune convolutional filters associated with unneeded classes.CaptorXcan reconfigure a pretrained full-class CNN model into class-specific lightweight models based on the visualization analysis of convolutional filters’ exclusive functionality for a single class. These lightweight models can be directly deployed to mobile devices without the retraining cost of traditional pruning-based reconfiguration. Furthermore, we can apply theCaptorXframework into a distributed collaboration setting. With dedicated local training regulation and collaborative aggregation schemes, the class-adaptive models on individual mobile devices can further contribute back to the central full-class model. Experiments on representative CNNs and image classification datasets show that,CaptorXcan reduce the CNN computation workload up to 50.22% and save 46.58% energy consumption for varied local devices, meanwhile improving accuracy for their targeted classes with better task focus. With our distributed collaboration paradigm,CaptorXalso provides further potential to enhance the central model accuracy, while reducing up to 37.58% communication cost compared to traditional distributed learning methods.

Deep neural networks (DNNs) achieved great cognitive performance at the expense of a considerable computation workload. To relieve the computational burden, many optimization works are developed to reduce the model redundancy by identifying and removing insignificant model components, such as weight sparsity and filter pruning methods. However, these works only evaluate model components’ static significance with parameter information, ignoring their dynamic interaction with external inputs. Specifically, due to the difference in per-input features, the model components’ significance can dynamically change and, thus, the static methods can only achieve suboptimal performance. Focusing on this aspect, we propose a dynamic DNN optimization framework in this work. Based on the neural network attention mechanism, we propose a comprehensive dynamic optimization framework, including 1) testing-phase dynamic feature map pruning; 2) training-phase optimization by training with targeted dropout; and 3) deployment-phase one-for-all (OFA) model adaptability enhancement. By providing a holistic dynamic testing, training, and deployment co-optimization framework, our work has the following benefits: first, it can accurately identify and aggressively remove per-input feature redundancy by considering the model-input interaction and involving the channel/column-wise pruning flexibility; meanwhile, the training-testing co-optimization favors the dynamic pruning and helps maintain the model accuracy even with a very high feature pruning ratio. Finally, the deployment enhancement provides one unified OFA model to support full-spectrum feature sparsity ratios. The unified model can be dynamically reconfigured to meet different resource budgets without any retraining cost, and thus provide significant deployment flexibility. Extensive experiments show that our method could bring 37.4%–54.5% floating-point operations reduction with negligible accuracy drop on various test benchmarks. Meanwhile, the OFA deployment optimization enables us to use one model to support at most ten different resource constraints without any retraining cost.

The accelerated convergence of digital and real-world lifestyles has imposed unprecedented demands on today’s wireless network architectures, as it is highly desirable for such architectures to support wireless devices everywhere with high capacity and minimal signaling overhead. Conventional architectures, such as cellular architectures, are not able to satisfy these requirements simultaneously, and are thus no longer suitable for the future era. In this paper, we propose a capacity-centric (C2) architecture for future wireless networking. It is designed based on the principles of maximizing the number of non-overlapping clusters with the average cluster capacity guaranteed to be higher than a certain threshold, and thus provides a flexible way to balance the capacity requirement against the signaling overhead. Our analytical results reveal that C2 has superior generality, wherein both the cellular and the fully coordinated architectures can be viewed as its extreme cases. Simulation results show that the average capacity of C2 is at least three times higher compared to that of the cellular architecture. More importantly, different from the widely adopted conventional wisdom that base-station distributions dominate architecture designs, we find that the C2 architecture is not over-reliant on base-station distributions, and instead the user-side information plays a vital role and cannot be ignored.

As Federated Learning (FL) has been widely used for collaborative training, a considerable computational straggler issue emerged: when FL deploys identical neural network models to heterogeneous devices, the ones with weak computational capacities, referred to as stragglers, may significantly delay the synchronous parameter aggregation. Although discarding stragglers from the collaboration can relieve this issue to a certain extent, stragglers may keep unique and critical information learned from the non-identical dataset, and directly discarding will harm the overall collaboration performance. Therefore, in this paper, we propose Helios – a heterogeneity-aware FL framework to tackle the straggler issue. Helios identifies individual devices’ heterogeneous training capability, and therefore the expected neural network model training volumes regarding the collaborative training pace. For straggling devices, a “softtraining” method is proposed to dynamically compress the original identical training model into the expected volume through a rotated neuron training approach. With extensive algorithm analysis and optimization schemes, stragglers can be accelerated while retaining the convergence for local training as well as federated collaboration. Experiments show that Helios can provide up to $2.5\times$ training acceleration and maximum 4.64% convergence accuracy improvement in various collaboration settings.

Jianguo Zhang, Trung Bui, Seunghyun Yoon, Xiang Chen, Zhiwei Liu, Congying Xia, Quan Hung Tran, Walter Chang, Philip Yu. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021.

With the fast development of deep neural networks (DNNs), many real-world applications are adopting multiple models to conduct compound tasks, such as co-running classification, detection, and segmentation models on autonomous vehicles. Such multi-tenant DNN inference cases greatly exacerbate the computational complexity and call for comprehensive collaboration for graph-level operator scheduling, runtime-level resource awareness, as well as hardware scheduler support. However, the current scheduling support for such multi-tenant inference is still relatively backward. In this work, we propose a resource-aware scheduling framework for efficient multi-tenant DNN inference on GPU, which automatically coordinates DNN computing in different execution levels. Leveraging the unified scheduling intermediate representation and the automated ML-based searching algorithm, optimal schedules could be generated to wisely adjust model concurrency and interleave DNN model operators, maintaining a continuously balanced resource utilization across the entire inference process, and eventually improving the runtime efficiency. Experiments show that we could consistently achieve$1.3\times\sim 1.7\times$speed-up, comparing to regular DNN runtime libraries (e.g., CuDNN, TVM) and particular concurrent scheduling methods (e.g., NVIDIA Multi-Stream).

We describe a novel algorithm for an open-domain trending hashtag recommendation task using zero-shot hashtag prediction in an online learning paradigm. Our method utilizes joint representation learning of latent embeddings for features extracted from long-form videos and semantic embeddings of hashtags trending on social platforms. In particular, we apply graph convolutional networks to a link prediction task using videos and hashtags as nodes in a heterogeneous graph. Comparing it to the existing models for closely related tasks, we demonstrate state-of-the-art results in trending hashtag recommendations for videos. The architecture is designed to be modular in a plug-and-play fashion to enable quick and easy incorporation of the latest advances in natural language understanding and image and video processing, with a practical view to its implementation in a real-time, online setting.

Federated learning learns from scattered data by fusing collaborative models from local nodes. However, conventional coordinate-based model averaging by FedAvg ignored the random information encoded per parameter and may suffer from structural feature misalignment. In this work, we propose Fed2, a feature-aligned federated learning framework to resolve this issue by establishing a firm structure-feature alignment across the collaborative models. Fed2 is composed of two major designs: First, we design a feature-oriented model structure adaptation method to ensure explicit feature allocation in different neural network structures. Applying the structure adaptation to collaborative models, matchable structures with similar feature information can be initialized at the very early training stage. During the federated learning process, we then propose a feature paired averaging scheme to guarantee aligned feature distribution and maintain no feature fusion conflicts under either IID or non-IID scenarios. Eventually, Fed2 could effectively enhance the federated learning convergence performance under extensive homo- and heterogeneous settings, providing excellent convergence speed, accuracy, and computation/communication efficiency.

This paper studies the decentralized optimization and learning problem where multiple interconnected agents aim to learn an optimal decision function defined over a reproducing kernel Hilbert space by jointly minimizing a global objective function, with access to their own locally observed dataset. As a non-parametric approach, kernel learning faces a major challenge in distributed implementation: the decision variables of local objective functions are data-dependent and thus cannot be optimized under the decentralized consensus framework without any raw data exchange among agents. To circumvent this major challenge, we leverage the random feature (RF) approximation approach to enable consensus on the function modeled in the RF space by data-independent parameters across different agents. We then design an iterative algorithm, termed DKLA, for fast-convergent implementation via ADMM. Based on DKLA, we further develop a communication-censored kernel learning (COKE) algorithm that reduces the communication load of DKLA by preventing an agent from transmitting at every iteration unless its local updates are deemed informative. Theoretical results in terms of linear convergence guarantee and generalization performance analysis of DKLA and COKE are provided. Comprehensive tests on both synthetic and real datasets are conducted to verify the communication efficiency and learning effectiveness of COKE.

With the advance of unmanned aerial vehicles (UAVs) and low earth orbit (LEO) satellites, the integration of space, air and ground networks has become a potential solution to the beyond fifth generation (B5G) Internet of remote things (IoRT) networks. However, due to the network heterogeneity and the high mobility of UAVs and LEOs, how to design an efficient UAV-LEO integrated data collection scheme without infrastructure support is very challenging. In this paper, we investigate the resource allocation problem for a two-hop uplink UAV-LEO integrated data collection for the B5G IoRT networks, where numerous UAVs gather data from IoT devices and transmit the IoT data to LEO satellites. In order to maximize the data gathering efficiency in the IoT-UAV data gathering process, we study the bandwidth allocation of IoT devices and the 3-dimensional (3D) trajectory design of UAVs. In the UAV-LEO data transmission process, we jointly optimize the transmit powers of UAVs and the selections of LEO satellites for the total uploaded data amount and the energy consumption of UAVs. Considering the relay role and the cache capacity limitations of UAVs, we merge the optimizations of IoT-UAV data gathering and UAV-LEO data transmission into an integrated optimization problem, which is solved with the aid of the successive convex approximation (SCA) and the block coordinate descent (BCD) techniques. Simulation results demonstrate that the proposed scheme achieves better performance than the benchmark algorithms in terms of both energy consumption and total upload data amount.

Although convolutional neural networks (CNNs) have been widely applied in various cognitive applications, they are still very computationally intensive for resource-constrained mobile systems. To reduce the resource consumption of CNN computation, many optimization works have been proposed for mobile CNN deployment. However, most works are merely targeting CNN model compression from the perspective of parameter size or model structure, ignoring different resource constraints in mobile systems with respect to memory, energy, and real-time requirement. Moreover, previous works take accuracy as their primary consideration, requiring a time-costing retraining process to compensate the inference accuracy loss after compression. To address these issues, we propose DiReCtX-a dynamic resource-aware CNN model reconfiguration framework. DiReCtX is based on a set of accurate CNN profiling models for different resource consumption and inference accuracy estimation. With manageable consumption/accuracy tradeoffs, DiReCtX can reconfigure a CNN model to meet distinct resource constraint types and levels with expected inference performance maintained. To further achieve fast model reconfiguration in real-time, improved CNN model pruning and its corresponding accuracy tuning strategies are also proposed in DiReCtX. The experiments show that the proposed CNN profiling models can achieve 94.6% and 97.1% accuracy for CNN model resource consumption and inference accuracy estimation. Meanwhile, the proposed reconfiguration scheme of DiReCtX can achieve at most 44.44% computation acceleration, 31.69% memory reduction, and 32.39% energy saving, respectively. On field-tests with state-of-the-art smartphones, DiReCtX can adapt CNN models to various resource constraints in mobile application scenarios with optimal real-time performance.

In recent years, the neural network (NN) has shown its great potential in image recognition tasks of autonomous driving systems, such as traffic sign recognition, pedestrian detection, etc. However, theoretically well-trained NNs usually fail their performance when facing real-world scenarios. For example, adverse real-world conditions, e.g., bad weather and lighting conditions, can introduce different physical variations and cause considerable accuracy degradation. As for now, the generalization capability of NNs is still one of the most critical challenges for the autonomous driving system. To facilitate the robust image recognition tasks, in this work, we build the RobuTS dataset: a comprehensive Robust Traffic Sign Recognition dataset, which includes images with different environmental variations, e.g., rain, fog, darkening, and blurring. Then to enhance the NN's generalization capability, we propose two generalization-enhanced training schemes: 1) REIN for robust training without data in adverse scenarios and 2) Self-Teaching (ST) for robust training with unlabeled adverse data. The great advantages of such two training schemes are they are data-free (REIN) and label-free (ST), thus effectively reducing the huge human efforts/cost of on-road driving data collection, as well as the expensive manual data annotation. We conduct extensive experiments to validate our methods' performance on both classification and detection tasks. For classification tasks, our proposed training algorithms could consistently improve model performance by +15%-25% (REIN) and +16%-30% (ST) in all adverse scenarios of our RobuTS datasets. For detection tasks, our ST could also improve the detector's performance by +10.1 mean average precision (mAP) on Foggy-Cityscapes, outperforming previous state-of-the-art works by +2.2 mAP.

The memristor crossbar array has emerged as an intrinsically suitable matrix computation and low-power acceleration framework for DNN applications. Many techniques such as memristor-based weight pruning and memristor-based quantization have been studied. However, the high accuracy solution for the above techniques is still waiting for unraveling. In this paper, we propose a memristor-based DNN framework which combines both structured weight pruning and quantization by incorporating ADMM algorithm for better pruning and quantization performance. We also discover the non-optimality of the ADMM solution in weight pruning and the unused data path in a structured pruned model. We design a software-hardware co-optimization framework which contains the first proposed Network Purification and Unused Path Removal algorithms targeting on post-processing a structured pruned model after ADMM steps. By taking memristor hardware constraints into our whole framework, we achieve extreme high compression rate with minimum accuracy loss. For quantizing structured pruned model, our framework achieves nearly no accuracy loss after quantizing weights to 8-bit memristor weight representation. We share our models at anonymous link https://bit.ly/2VnMUy0.

Recently, adversarial attacks can be applied to the physical world, causing practical issues to various Convolutional Neural Networks (CNNs) powered applications. Most existing physical adversarial attack defense works only focus on eliminating explicit perturbation patterns from inputs, ignoring interpretation to CNN's intrinsic vulnerability. Therefore, they lack expected versatility to different attacks and thereby depend on considerable data processing costs. In this paper, we propose LanCe - a comprehensive and lightweight CNN defense methodology against different physical adversarial attacks. By interpreting CNN's vulnerability, we find that non-semantic adversarial perturbations can activate CNN with significantly abnormal activations and even overwhelm other semantic input patterns' activations. We improve the CNN recognition process by adding a self-verification stage to detect the potential adversarial input with only one CNN inference cost. Based on the detection result, we further propose a data recovery methodology to defend the physical adversarial attacks. We apply such defense methodology into both image and audio CNN recognition scenarios and analyze the computational complexity for each scenario, respectively. Experiments show that our methodology can achieve an average 91% successful rate for attack detection and 89% accuracy recovery. Moreover, it is at most 3× faster compared with the state-of-the-art defense methods, making it feasible to resource-constrained embedded systems, such as mobile devices.

Convolutional Neural Networks (CNNs) achieved great cognitive performance at the expense of considerable computation load. To relieve the computation load, many optimization works are developed to reduce the model redundancy by identifying and removing insignificant model components, such as weight sparsity and filter pruning. However, these works only evaluate model components’ static significance with internal parameter information, ignoring their dynamic interaction with external inputs. With per-input feature activation, the model component significance can dynamically change, and thus the static methods can only achieve sub-optimal results. Therefore, we propose a dynamic CNN optimization framework in this work. Based on the neural network attention mechanism, we propose a comprehensive dynamic optimization framework including (1) testing-phase channel and column feature map pruning, as well as (2) training-phase optimization by targeted dropout. Such a dynamic optimization framework has several benefits: (1) First, it can accurately identify and aggressively remove per-input feature redundancy with considering the model-input interaction; (2) Meanwhile, it can maximally remove the feature map redundancy in various dimensions thanks to the multi-dimension flexibility; (3) The training-testing co-optimization favors the dynamic pruning and helps maintain the model accuracy even with very high feature pruning ratio. Extensive experiments show that our method could bring 37.4%∼54.5% FLOPs reduction with negligible accuracy drop on various of test networks.

Recently Convolutional Neural Networks (CNNs) are widely applied into novel intelligent applications and systems. However, the CNN computation performance is significantly hindered by its computation flow, which computes the model structure sequentially by layers with massive convolution operations. Such a layer-wise sequential computation flow can cause certain performance issues, such as resource under-utilization, huge memory overhead, etc. To solve these problems, we propose a novel CNN structural decoupling method, which could decouple CNN models into "critical paths" and eliminate the inter-layer data dependency. Based on this method, we redefine the CNN computation flow into parallel and cascade computing paradigms, which can significantly enhance the CNN computation performance with both multi-core and single-core CPU processors. Experiments show that, our DC-CNN framework could reduce 24% to 33% latency on multi-core CPUs for CIFAR and ImageNet. On small-capacity mobile platforms, cascade computing could reduce the latency by average 24% on ImageNet and 42% on CIFAR10. Meanwhile, the memory reduction could also reach average 21% and 64%, respectively.

Recent progress in deep learning techniques enabled collaborative edge training, which usually deploys identical neural network models globally on multiple devices for aggregating parameter updates over distributed data collection. However, as more and more heterogeneous edge devices are involved in practical training, the identical model deployment over collaborative edge devices cannot be guaranteed: On one hand, the weak edge devices with less computation resources may not catch up stronger ones' training progress, and appropriate local model training customization is necessary to balance the collaboration. On the other hand, a particular local edge device may have specific learning task preference, while the global identical model would exceed the practical local demand and cause unnecessary computation cost. Therefore, we explored the collaborative learning with heterogeneous convolutional neural networks (CNNs) in this work, expecting to address aforementioned real problems. Specifically, we proposed a novel decentralized collaborative training method by decoupling a training target CNN model into independently trainable sub-models correspond to a sub-set of learning tasks for each edge device. After sub-models are well-trained on edge nodes, the model parameters for individual learning tasks can be harvested from local models on every edge device and ensemble the global training model back to a single piece. Experiments demonstrate that, for the AlexNet and VGG on the CIFAR10, CIFAR100 and KWS dataset, our decentralized training method can save up to 11.8× less computation load while achieve central sever test accuracy.

This paper gives an overview of our ongoing work on the design space exploration of efficient deep neural networks (DNNs), specifically on the novel optimization perspectives that past work have mainly overlooked. We cover two complementary aspects of efficient DNN design: (1) static architecture design efficiency and (2) dynamic model execution efficiency. In the static architecture design, one of the major challenges of NAS is the low search efficiency. Different with current mainstream efficient search algorithm optimization, we identify the new perspective in efficient search space design. In the dynamic model execution, current major optimization methods still target at the model structure redundancy, e.g., weight/filter pruning, connection pruning, etc. We instead identify the new dimension of DNN feature map redundancy. By showcasing such new perspectives, further advantages could be potentially attained by integrating both current optimizations and our new perspectives.

Resistive random-access memory (ReRAM) based accelerators have been widely studied to achieve efficient neural network computing in speed and energy. Neural network optimization algorithms such as sparsity are developed to achieve efficient neural network computing on traditional computer architectures such as CPU and GPU. However, such computing efficiency improvement is hindered when deploying these algorithms on the ReRAM-based accelerator because of its unique crossbar-structural computations. And a specific algorithm and hardware co-optimization for the ReRAM-based architecture is still in a lack. In this work, we propose an efficient neural network computing framework that is specialized for the crossbar-structural computations on the ReRAM-based accelerators. The proposed framework includes a crossbar specific feature map pruning and an adaptive neural network deployment. Experimental results show our design can improve the computing accuracy by 9.1% compared with the state-of-the-art sparse neural networks. Based on a famous ReRAM-based DNN accelerator, the proposed framework demonstrates up to 1.4× speedup, 4.3× power efficiency, and 4.4× area saving.

Predicting user engagement (e.g., click-through rate, conversion rate) on the display ads plays a critical role in delivering the right ad to the right user in online advertising. Existing techniques spanning Logistic Regression to Factorization Machines and their derivatives, focus on modeling the interactions among handcrafted features to predict the user engagement. Little attention has been paid on how the ad fits with the context (e.g., hosted webpage, user demographics). In this paper, we propose to include the metadata feature, which captures the visual appearance of the ad, in the user engagement prediction task. In particular, given a data sample, we combine both the basic context features, which have been widely used in existing prediction models, and the metadata feature, which is extracted from the ad using a state-of-the-art deep learning framework, to predict user engagement. To demonstrate the effectiveness of the proposed metadata feature, we compare the performance of the widely used prediction models before and after integrating the metadata feature. Our experimental results on a real-world dataset demonstrate that the metadata feature is able to further improve the prediction performance.

Nowadays, the evolution of deep learning and cloud service significantly promotes neural network based mobile applications. Although intelligent and prolific, those applications still lack certain flexibility: For classification tasks, neural networks are generally trained online with vast classification targets to cover various utilization contexts. However, only partial classes are practically tested due to individual mobile user preference and application specificity. Thus the unneeded classes cause considerable computation and communication cost. In this work, we propose CAPTOR - a class-level reconfiguration framework for Convolutional Neural Networks (CNNs). By identifying the class activation preference of convolutional filters through feature interest visualization and gradient analysis, CAPTOR can effectively cluster and adaptively prune the filters associated with unneeded classes. Therefore, CAPTOR enables class-level CNN reconfiguration for network model compression and local deployment on mobile devices. Experiment shows that, CAPTOR can reduce computation load for VGG-16 by up to 40.5% and 37.9% energy consumption with ignored loss of accuracy. For AlexNet, CAPTOR also reduces computation load by up to 42.8% and 37.6% energy consumption with less than 3% loss in accuracy.

Recently, the machine learning technologies have been widely used in cognitive applications such as Automatic Speech Recognition (ASR) and Image Recognition (IR). Unfortunately, these techniques have been massively used in unauthorized audio/image data analysis, causing serious privacy leakage. To address this issue, we propose HAMPER in this work, which is a data encryption framework that protects the audio/image data from unauthorized ASR/IR analysis. Leveraging machine learning models' vulnerability to adversarial examples, HAMPER encrypt the audio/image data with adversarial noises to perturb the recognition results of ASR/IR systems. To deploy the proposed framework in extensive platforms (e.g. mobile devices), HAMPER also take into consideration of computation efficiency, perturbation transferability, as well as data attribute configuration. Therefore, rather than focusing on the high-level machine learning models, HAMPER generates adversarial examples from the low-level features. Taking advantage of the light computation load, fundamental impact, and direct configurability of the low-level features, the generated adversarial examples can efficiently and effectively affect the whole ASR/IR systems. Experiment results show that, HAMPER can effectively perturb the unauthorized ASR/IR analysis with 85% Word-Error-Rate (WER) and 83% Image-Error-Rate (IER) respectively. Also, HAMPER achieves faster processing speed with 1.5X speedup for image encryption and even 26X in audio, comparing to the state-of-the-art methods. Moreover, HAMPER achieves strong transferability and configures adversarial examples with desired attributes for better scenario adaptation.

In recent years, neural network has shown its great potential in autonomous driving systems. However, the theoretically well-train neural networks usually fail their performance when facing real-world examples with unexpected physical variations. As the current neural networks still suffer from limited generalization ability, those unexpected variations would cause considerable accuracy degradation and critical safety issues. Therefore, the generalization ability of neural networks becomes one of the most critical challenges for autonomous driving system design. In this work, we propose a robust training method to enhance neural network's generalization ability in various practical autonomous driving scenarios. Based on detailed practical variation modeling and neural network generation ability analysis, the proposed training method could consistently improve model classification accuracy by at most 25% in various scenarios (e.g. raining/fogy, dark lighting, and camera discrepancy). Even with adversarial corner cases, our model could still achieve at most 40% accuracy improvement over natural model.

At present, object recognition studies are mostly conducted in a closed lab setting with classes in test phase typically in training phase. However, real-world problem are far more challenging because: i)~new classes unseen in the training phase can appear when predicting; ii)~discriminative features need to evolve when new classes emerge in real time; and iii)~instances in new classes may not follow the "independent and identically distributed" (iid) assumption. Most existing work only aims to detect the unknown classes and is incapable of continuing to learn newer classes. Although a few methods consider both detecting and including new classes, all are based on the predefined handcrafted features that cannot evolve and are out-of-date for characterizing emerging classes. Thus, to address the above challenges, we propose a novel generic end-to-end framework consisting of a dynamic cascade of classifiers that incrementally learn their dynamic and inherent features. The proposed method injects dynamic elements into the system by detecting instances from unknown classes, while at the same time incrementally updating the model to include the new classes. The resulting cascade tree grows by adding a new leaf node classifier once a new class is detected, and the discriminative features are updated via an end-to-end learning strategy. Experiments on two real-world datasets demonstrate that our proposed method outperforms existing state-of-the-art methods.

Although the Deep Neural Network (DNN) technique has been widely applied in various applications, the DNN-based applications are still too computationally intensive for the resource-constrained mobile devices. Many works have been proposed to optimize the DNN computation performance, but most of them are limited in an algorithmic perspective, ignoring certain computing issues in practical deployment. To achieve the comprehensive DNN performance enhancement in practice, the expected DNN optimization works should closely cooperate with specific hardware and system constraints (i.e. computation capacity, energy cost, memory occupancy, and inference latency). Therefore, in this work, we propose ReForm -- a resource-aware DNN optimization framework. Through thorough mobile DNN computing analysis and innovative model reconfiguration schemes (i.e. ADMM based static model fine-tuning, dynamically selective computing), ReForm can efficiently and effectively reconfigure a pre-trained DNN model for practical mobile deployment with regards to various static and dynamic computation resource constraints. Experiments show that ReForm has ~3.5× faster optimization speed than state-of-the-art resource-aware optimization method. Also, ReForm can effective reconfigure a DNN model to different mobile devices with distinct resource constraints. Moreover, ReForm achieves satisfying computation cost reduction with ignorable accuracy drop in both static and dynamic computing scenarios (at most 18% workload, 16.23% latency, 48.63% memory, and 21.5% energy enhancement).

Benefited from recent artificial intelligence evolution, Automatic Speech Recognition (ASR) technology has achieved enormous performance improvement and wider application. Unfortunately, ASR is also heavily leveraged by speech eavesdropping, where ASR is used to translate large volume of intercepted vocal speech into text content, causing considerable information leakage. In this work, we propose MASKER -- a mobile security enhancement solution to protect the mobile speech data from ASR in eavesdropping. By identifying ASR models' ubiquitous vulnerability, MASKER is designed to generate human imperceptible adversarial noises into the real-time speech on the mobile device (e.g. phone call and voice message). Even the speech data is exposed to eavesdropping during data transmission, the adversarial noises can effectively perturb the ASR process with significant Word Error Rate (WER). Meanwhile, MASKER is further optimized for mobile user perception quality and enhanced for environmental noises adaptation. Moreover, MASKER has outstanding computation efficiency for mobile system integration. Experiments show that, MASKER can achieve security enhancement with an average WER of 84.55% for ASR perturbation, 32% noise reduction for user perception quality and 16× faster processing speed compared to the state-of-the-art method.

Recently, adversarial deception becomes one of the most considerable threats to deep neural networks. However, compared to extensive research in new designs of various adversarial attacks and defenses, the neural networks' intrinsic robustness property is still lack of thorough investigation. This work aims to qualitatively interpret the adversarial attack and defense mechanisms through loss visualization, and establish a quantitative metric to evaluate the model's intrinsic robustness. The proposed robustness metric identifies the upper bound of a model's prediction divergence in the given domain and thus indicates whether the model can maintain a stable prediction. With extensive experiments, our metric demonstrates several advantages over conventional testing accuracy based robustness estimation: (1) it provides a uniformed evaluation to models with different structures and parameter scales; (2) it over-performs conventional accuracy based robustness evaluation and provides a more reliable evaluation that is invariant to different test settings; (3) it can be fast generated without considerable testing cost.

Alternating Direction Method of Multipliers (ADMM) has been used successfully in many conventional machine learning applications and is considered to be a useful alternative to Stochastic Gradient Descent (SGD) as a deep learning optimizer. However, as an emerging domain, several challenges remain, including 1) The lack of global convergence guarantees, 2) Slow convergence towards solutions, and 3) Cubic time complexity with regard to feature dimensions. In this paper, we propose a novel optimization framework for deep learning via ADMM (dlADMM) to address these challenges simultaneously. The parameters in each layer are updated backward and then forward so that the parameter information in each layer is exchanged efficiently. The time complexity is reduced from cubic to quadratic in (latent) feature dimensions via a dedicated algorithm design for subproblems that enhances them utilizing iterative quadratic approximations and backtracking. Finally, we provide the first proof of global convergence for an ADMM-based method (dlADMM) in a deep neural network problem under mild conditions. Experiments on benchmark datasets demonstrated that our proposed dlADMM algorithm outperforms most of the comparison methods.

Live streaming has become a new form of entertainment, which attracts hundreds of millions of users worldwide. The huge amount of multimedia data in live streaming platforms creates tremendous opportunities for online advertising. However, existing state-of-the-art video advertising strategies (e.g., pre-roll and contextual mid-roll advertising) that rely on analyzing the whole video, are not applicable to live streaming videos. This paper describes a novel monetization framework, named LiveSense, for live streaming videos, which is able to display a contextually relevant ad at a suitable timestamp in a non-intrusive way. Specifically, given a live streaming video, we first employ a deep neural network to determine whether the current moment is appropriate for displaying an ad using the historical streaming data. Then, we detect a set of candidate ad insertion areas by incorporating image saliency, background map, and location priorities, so that the ad is displayed over the non-important area. We introduce three types of relevance metrics including textual relevance, global visual relevance and local visual relevance to select the contextually relevant ad. To minimize user intrusiveness, we initially display the ad at a non-important area. If the user is interested in the ad, we will show the ad in an overlaid window with a translucent background. Empirical evaluation on a real-world dataset demonstrates that our proposed framework is able to effectively display ads in live streaming videos while maintaining users' online experience.

Although Convolutional Neural Networks (CNNs) have been widely applied in various applications, their deployment in resource-constrained mobile systems remains a significant concern. To overcome the computation resource constraints, such as limited memory and energy capacity, many works are proposed for mobile CNN optimization. However, most of them lack a comprehensive modeling analysis of the CNN computation consumption and merely focus on static optimization schemes regardless of different mobile computation scenarios. In this work, we proposed DiReCt -- a resource-aware CNN reconfiguration system. Leveraging accurate CNN computation consumption modeling and mobile resource constraint analysis, DiReCt can reconfigure a CNN with different accuracy and resource consumption levels to adapt to various mobile computation scenarios. The experiment results show that: the proposed computation consumption models in DiReCt can well estimate the CNN computation consumption with 94.1% accuracy, and DiReCt achieves at most 34.9% computation acceleration, 52.7% memory reduction, and 27.1% energy saving. Eventually, DiReCt can effectively adapt CNNs to dynamic mobile usage scenarios for optimal performance.

Although Deep Neural Networks (DNN) are ubiquitously utilized in many applications, it is generally difficult to deploy DNNs on resource-constrained devices, e.g., mobile platforms. Some existing attempts mainly focus on client-server computing paradigm or DNN model compression, which require either infrastructure supports or special training phases, respectively. In this work, we propose MoDNN - a local distributed mobile computing system for DNN applications. MoDNN can partition already trained DNN models onto several mobile devices to accelerate DNN computations by alleviating device-level computing cost and memory usage. TWo model partition schemes are also designed to minimize non-parallel data delivery time, including both wakeup time and transmission time. Experimental results show that when the number of worker nodes increases from 2 to 4, MoDNN can accelerate the DNN computation by 2.17-4.28 X. Besides the parallel execution, the performance speedup also partially comes from the reduction of the data delivery time, e.g., 30.02% w.r.t. conventional 2D-grids partition.

Neural networks hold a critical domain in machine learning algorithms because of their self-adaptiveness and state-of-the-art performance. Before the testing (inference) phases in practical use, sophisticated training (learning) phases are required, calling for efficient training methods with higher accuracy and shorter converging time. Many existing studies focus on the training optimization on high-performance servers or computing clusters, e.g. GPU clusters. However, training neural networks on resource-constrained devices, e.g. mobile platforms, is an important research topic barely touched. In this paper, we implement AdaLearner-an adaptive distributed mobile learning system for neural networks that trains a single network with heterogenous mobile resources under the same local network in parallel. To exploit the potential of our system, we adapt neural networks training phase to mobile device-wise resources and fiercely decrease the transmission overhead for better system scalability. On three representative neural network structures trained from two image classification datasets, AdaLearner boosts the training phase significantly. For example, on LeNet, 1.75-3.37× speedup is achieved when increasing the worker nodes from 2 to 8, thanks to the achieved high execution parallelism and excellent scalability.

Deep Neural Networks (DNNs) are pervasively used in a significant number of applications and platforms. To enhance the execution efficiency of large-scale DNNs, previous attempts focus mainly on client-server paradigms, relying on powerful external infrastructure, or model compression, with complicated pre-processing phases. Though effective, these methods overlook the optimization of DNNs on distributed mobile devices. In this work, we design and implement MeDNN, a local distributed mobile computing system with enhanced partitioning and deployment tailored for large-scale DNNs. In MeDNN, we first propose Greedy Two Dimensional Partition (GTDP), which can adaptively partition DNN models onto several mobile devices w.r.t. individual resource constraints. We also propose Structured Model Compact Deployment (SMCD), a mobile-friendly compression scheme which utilizes a structured sparsity pruning technique to further accelerate DNN execution. Experimental results show that, GTDP can accelerate the original DNN execution time by 1.86-2.44x with 2-4 worker nodes. By utilizing SMCD, 26.5% of additional computing time and 14.2% of extra communication time are saved, on average, with negligible effect on the model accuracy.

Convolutional Neural Networks (CNNs) have been widely investigated as some of the most promising solution for various computer vision tasks. However, CNNs introduce massive computing overhead due to their complex network computing flow, resulting in significantly reduced applicability and performance, especially in the mobile devices. Various optimization schemes have been proposed mainly based on both model compression and stacked external computing resources. While these schemes have been proven effective, methods which take into account mobile-specific context-aware optimization approaches have been largely overlooked. One such opportunity is the feasible CNN computing flow simplification to the under-test objects with distinguish features, which can be efficiently pre-analyzed inside the mobile sensor system. Hence, we propose VoCaM, a visualization oriented CNN acceleration framework on mobile devices for image classification tasks. VoCaM takes advantage of the mobile camera system, where the comprehensive pre-analysis can be conducted to reveal the color composition of the under-test images without incurring any additional overhead. Also, the visualization analysis of VoCaM reveals that, certain color-specific filters may have very trivial result impact when the under-test images have mismatching primary color components. Then a set of approximate computing methods is applied to these insignificant filters to replace the intensive convolutional operation, and greatly accelerate the computing process. With ignorable overhead, VoCaM can significantly optimize the computation load of the convolutional layers, with very small impact on the overall classification accuracy.

Displaying banner advertisements (in short, ads) on webpages has usually been discussed as an Internet economics topic where a publisher uses auction models to sell an online user's page view to advertisers and the one with the highest bid can have her ad displayed to the user. This is also called real-time bidding (RTB) and the ad displaying process ensures that the publisher's benefit is maximized or there is an equilibrium in ad auctions. However, the benefits of the other two stakeholders - the advertiser and the user - have been rarely discussed. In this paper, we propose a two-stage computational framework that selects a banner ad based on the optimized trade-offs among all stakeholders. The first stage is still auction based and the second stage re-ranks ads by considering the benefits of all stakeholders. Our metric variables are: the publisher's revenue, the advertiser's utility, the ad memorability, the ad click-through rate (CTR), the contextual relevance, and the visual saliency. To the best of our knowledge, this is the first work that optimizes trade-offs among all stakeholders in RTB by incorporating multimedia metrics. An algorithm is also proposed to determine the optimal weights of the metric variables. We use both ad auction datasets and multimedia datasets to validate the proposed framework. Our experimental results show that the publisher can significantly improve the other stakeholders' benefits by slightly reducing her revenue in the short-term. In the long run, advertisers and users will be more engaged, the increased demand of advertising and the increased supply of page views can then boost the publisher's revenue.

Wrist-worn wearable fitness devices, such as FitBit and Apple Watch, have become popular in recent years. Runners can use the GPS embedded in these wearable devices to log the route taken during their exercise, providing vital feedback on pace and distance traveled. Unfortunately, continuous polling for GPS data results in a significant adverse impact on device battery life, e.g., many flagship wearables need to be charged as frequently as every two days or even less. In this work, we propose Footfall - an intelligent GPS scheduler that can utilize data from alternative sensors on a device to greatly reduce GPS utilization while still maintaining minimum location accuracy. Compared to existing implementations, Footfall system can achieve on average 97% reduction in total power consumption, while only inducing 7% discrepancy in location accuracy, which is sufficient for the targeted applications.

In the era of network service, the user authentication becomes more indispensable but also vulnerable. Traditional user verification approaches such as PIN or pattern lock suffer from easy hacking and replica, motivating the research on many new approaches like gesture-based security. Compare to traditional authentications, the gesture-based security utilizes the user interacts with the device as a dynamic authentication pattern in real-time, offering higher complexity and better reliability. However, gesture-based security still lacks sufficient research on data sampling and preprocessing techniques on classification accuracy. In this work, we develop SlowMo, a novel gesture security technique for user classification in low sampling-rate environments. The proposed algorithm provides maximum classification accuracy at a sampling rate of 4Hz with extreme low power consumption suggesting a more capable adaptation to the security environment. It can achieve classification accuracy as high as 89% with power consumption negligible to the user.

Even with the adoption of the latest OLED technology, the display panel remains one of the most power-hungry components in smartphones. Existing attempts for OLED power optimization have mainly focused on modifying the content that is shown on the display during the playback phase, requiring significant overhead in terms of image analysis and modification. While such methods are effective, they overlook opportunities present during the camera recording phase, where utilization of already determined camera parameters could reduce or eliminate the image processing overhead. Hence, we proposed MORPh, a cross-layer optimization system for OLED. We first analyze three fundamental parameters extracted from the smartphone camera and their impact on OLED power consumption and visual quality. We then define corresponding metrics to assess power optimization potentials, and propose a set of algorithms that optimize camera recording to be more OLED friendly. Finally, the proposed schemes are realized as part of a video recording and playback application on an existing Android smartphone. The experiments results indicate power saving of 7.3%~39.7%, and 20.3% on average while maintaining perceived visual quality.

Modern smartphone display system come equipped with powerful GPU's capable of rendering advanced 2D and 3D graphics. These GPU's make up a significant portion of the system power profile due to the high resolution and framerate of smartphone display. These display features are selected during the design phase of a smartphone and correspond to the capabilities of the human visual system (HVS). However, the level of detail observable by the HVS is not static and changes with user-smartphone distance. In this paper we propose Scope, a system which alters the rendering resolution and framerate on a smartphone to scale display rendering workload in response to real time changes in user-smartphone distance. We demonstrate a new method of measuring this distance in real-time which is able to minimize front-facing camera sampling through the use of sensor fusion techniques. The result is that Scope requires a power overhead of only 20mW on average as the front-facing camera need only be sampled 4 times per minute. We demonstrate that Scope is able to reduce smartphone power consumption up to 58% while retaining visual quality in most cases.

With superior advantages of better display quality and power efficiency, the latest OLED technology has achieved unprecedented popularity in the display screen market. However, the OLED remains one of the most power-hungry components in mobile devices. Various optimization schemes have been proposed based on the color-dependent power consumption feature of OLED pixels. These schemes mainly focus on color modification during the playback phase and require significant overhead in terms of frame analysis and real-time modification. While such schemes are effective, the power saving opportunities during the camera recording phase are overlooked. To further enhance the power optimization, the camera parameters during the recording phase could be effectively utilized to reduce or eliminate the optimization overhead. Hence, we proposed MORPh, a cross-layer optimization system for OLED display in the smartphones. We analyze three fundamental parameters of smartphone camera system and their impact on the OLED screen power consumption. We then define corresponding metrics to quantitatively assess each parameter's potential of power saving guidance. Finally, we develop a set of schemes and integrated them into a video recording and playback application on an existing Android smartphone. The experiments results indicate power saving of 7.3%∼39.7%, and 20.3% on average while maintaining perceived visual quality.

The adoption of the latest OLED (organic light emitting diode) technology does not change the fact that screen is still one of the most energy-consuming modules in modern smartphones. In this work, we found that video streams from the same video category share many common power consumption features on OLED screens. Therefore, we are able to build a Hidden Markov Model (HMM) classifier to categorize videos based on OLED screen power characteristics. Using this HMM classifier, we propose a video classification based dynamic tone mapping (DTM) scheme, namely, DaTuM, to remap output color range and minimize the power-hungry color compositions on OLED screens for power saving. Experiment shows that DaTuM scheme averagely reduces OLED screen power by 17.8% with minimum display quality degradation. Compared to DTM scheme based on official category info provided by the video sources and one state-of-the-art scheme, DaTuM substantially enhances OLED screens' power efficiency and display quality controllability.

In this paper, we presented our recent characterization and analysis on the power consumption of smartphone radio components, including Wi-Fi, GPS and cellular (3G/4G) modules. Different from previous research that focused on the properties of single module under a limited number of usage scenarios, our works are performed on the statistically selected representative Apps with four generations of Samsung Galaxy series smartphones. We found that over the four characterized generations, the average power consumption of the smartphones with a connected Wi-Fi network on the selected Apps increases about 38.20%, indicating more and more energy-hungry designs. However, the raising of the power consumption is not mainly from the radio modules as the power efficiency of the Wi-Fi module indeed improves by about 21%. The 4G module consumes about 15.13% more power compared to the latest 3G module. But the power efficiency of 3G module improves by averagely 15.17% over the four generations. The GPS module also shows 25.80% power consumption improvement over the generations. We also show that the characterized power results can be combined with the real user traces for detailed daily energy usage estimation. Our research established an ethological and evolutionary power consumption trend of smartphones, which will guide the power optimization in the next-generation smartphone design.

Intelligent mobile devices have been widely serving in almost all aspects of everyday life, spanning from communication, web surfing, entertainment, to daily organizer. A large amount of sensitive and private information is stored on the mobile device, leading to severe data security concern. In this work, we propose a novel mobile user classification and authorization scheme based on the recognition of user's gesture. Compared to other security solutions like password, track pattern and finger print etc., our scheme can continuously evolve for better protection during the usage cycle of the mobile device. Besides the regular interactive screen and sensors of modern mobile devices, our scheme does not require any additional hardware supports.

Organic Light Emitting Diode (OLED) has emerged as the new generation display technique for mobile multimedia devices. Compared to existing technologies OLEDs are thinner, brighter, lighter, and cheaper. However, OLED panels are still the biggest contributor to the total power consumption of mobile devices. In this work, we proposed a fine-grained dynamic voltage scaling (FDVS) technique to reduce the OLED power. An OLED panel is partitioned into multiple display areas of which the supply voltage is adaptively adjusted based on the displayed content. A DVS-friendly OLED driver design is also proposed to enhance the color accuracy of the OLED pixels at the scaled supply voltage. Our experimental results show that compared to the existing global DVS technique, FDVS technique can achieve 25.9%∼43.1% more OLED power saving while maintaining a high image quality measured by Structural Similarity Index (SSIM=0.98). The further analysis shows shat FDVS technology can also effectively reduce the color remapping cost when color compensation is required to improve the image quality of an OLED panel working at a scaled voltage.

This paper developed a dynamic voltage scaling (DVS) technique for the power management of the OLED display on mobile devices in video streaming applications. An optimal voltage control scheme is proposed under input constraints. Fine-grained DVS technique is applied to maximize the power saving by leveraging the locality of the display content. The display quality is retained by monitoring structural-similarity-index (SSIM) during the optimization, subject to the hardware constraints like voltage regulator response time. Simulation results on four typical video test benchmarks show that the proposed technique saves 19.05%~49.05% OLED power on average while maintaining a high display quality (SSIM > 0.98) all the time. The power saving efficiency of the proposed technique varies at different display resolutions, refresh rates, and display contents.

OLED (Organic Light Emitting Diode) technology has already been adopted in many modern smart mobile devices, including cellphones, tablets, laptop etc. However, the power dissipation of displays in some applications like real-time video streaming, significantly limits the smart mobile devices' battery life and influences user experience. In this work, we applied a set of power management techniques that based on the dynamic voltage scaling (DVS) to minimize the power consumption of OLED display. Circuit inventions are also introduced to enable the local voltage scaling of AMOLED display panel while the human vision reception criteria can still be met. We then apply the DVS-based power management techniques to online video streaming, which is the most energy-hungry application of mobile devices: For any known type of mobile devices with AMOLED displays, the DVS power management scheme of a specific video stream can be pre-analyzed and shared on network, i.e., the Cloud or local party that provide the video. The DVS scheme can be downloaded in the real-time simultaneously when the video is broadcasted to the end users. By doing so, the power and computation overheads of DVS optimization are amortized among all the users, achieving a high service quality. Furthermore, the optimization can be executed locally (at user end) and shared by all the users.

OLED is becoming the main stream display for mobile devices. The process variations of thin-film transistors (TFT) and the aging degradation of OLED devices severely impact the display quality and the user experience on mobile devices throughout lifetime. In this paper, we quantitatively study the nonuniformity of OLED display panels incurred by the TFT variations and OLED cell aging effect. Furthermore, we develop a pixel level sensing circuit that detects and quantifies the nonuniformity condition and corresponding compensating technique. This proposed technique can be actively invoked based on various conditions with flexible configuration and minimal extra overhead, which is suitable to be integrated with mobile displays. The proposed technique's performance is simulated with different display contents. Experiments show that: for the proposed sensing circuit, the error rate for TFT process variation evaluation is only 4.7%~7.9%, and 0.29%~7.6% for OLED aging degradation. And for typical mobile display content, the compensation rate reaches 94.04%~100%. After applying the proposed techniques, the nonuniformity is unrecognizable and the OLED display panel's lifespan is highly extended.

In this article, we propose a data storage system with the emerging nonvolatile memory technologies used for the implantable electrocardiography (ECG) recorder. The proposed storage system can record the digitalized real-time ECG waveforms continuously inside the implantable device and export the stored data to external reader periodically to obtain a long-term backup. Spin transfer torque random access memory (STT-RAM) and spintronic memristor are selected as the storage elements for their nonvolatility, high density, high reliability, low power consumption, good scalability, and CMOS technology compatibility. The new read and write schemes of STT-RAM and spintronic memristors are presented and optimized to fit the specific application scenario. The tradeoffs among data accuracy, chip area, and read/write energy for the different technologies are thoroughly analyzed and compared. Our simulation results show the configuration with a data sampling rate (e.g., 128 Hz) and a quantization resolution (e.g., 12 bits) can record 18-hour real-time data within ~ 3.6-mm 2 chip area when the data storage is built with single-level cell (SLC) STT-RAMs. Daily energy consumption is 5.46 mJ. Utilizing the multilevel cell (MLC) STT-RAMs or the spintronic memristors as the storage elements can further reduce the chip area and decrease energy dissipation.