The growing misuse of Vision-Language Models (VLMs) has led providers to deploy multiple safeguards—alignment tuning, system prompt, and content moderation. Yet the real-world robustness of these defenses against adversarial attack remains underexplored. We introduce Multi-Faceted Attack (MFA), a framework that systematically uncovers general safety vulnerabilities in leading defense-equipped VLMs, including GPT-4o, Gemini-Pro, and LLaMA 4, etc. Central to MFA is the Attention-Transfer Attack (ATA), which conceals harmful instructions inside a meta task with competing objectives. We offer a theoretical perspective grounded in reward-hacking to explain why such an attack can succeed. To maximize cross-model transfer, we introduce a lightweight transfer-enhancement algorithm combined with a simple repetition strategy that jointly evades both input- and output-level filters—without any model-specific fine-tuning. We empirically show that adversarial images optimized for one vision encoder transfer broadly to unseen VLMs, indicating that shared visual representations create a cross-model safety vulnerability. Combined, MFA reaches a 58.5% overall attack success rate, consistently outperforming existing methods. Notably, on state-of-the-art commercial models, MFA achieves a 52.8% success rate, outperforming the second-best attack by 34%. These findings challenge the perceived robustness of current defensive mechanisms, systematically expose general safety loopholes within defense-equipped VLMs, and offer a practical probe for diagnosing and evaluating the safety of VLMs.

Lichao Wang, ZhaoXing Ren, Tianzhuo Yang, Jiaming Ji, Chi Harold Liu, Yaodong Yang, Juntao Dai. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

Deep learning models for natural language processing rely heavily on high-quality labeled datasets. However, existing labeling approaches often struggle to balance label quality with labeling cost. To address this challenge, we propose DALL, a text labeling framework that integrates data programming, active learning, and large language models. DALL introduces a structured specification that allows users and large language models to define labeling functions via configuration, rather than code. Active learning identifies informative instances for review, and the large language model analyzes these instances to help users correct labels and to refine or suggest labeling functions. We implement DALL as an interactive labeling system for text labeling tasks. Comparative, ablation, and usability studies demonstrate DALL’s efficiency, the effectiveness of its modules, and its usability.

Content-aware layout generation is crucial in poster design for automatically arranging layout elements. With the data scarcity problem, existing methods mainly employ retrieval augmentation or leverage Large Language Models (LLMs). However, these approaches still face persistent issues such as element overlap, misalignment, and high resource consumption (especially for LLMs). Additionally, these methods ignore and hardly handle layout generation based on pre-existing text canvases (i.e., text-rich images), which are common in real-world poster design. To address these issues, we propose LayoutGD, a graph-based diffusion method that aims to optimize overlap and misalignment while simultaneously enhancing performance on text-rich images. Our method represents all layout elements and image patches as independent nodes, constructs graphs based on specific topologies, and applies Graph Neural Networks (GNNs) to capture their high-dimensional spatial relationships. Furthermore, LayoutGD can process both text-clean and text-rich canvases in a unified framework, benefiting from our Enhance-Branch architecture. Extensive experiments demonstrate that our method achieves the state-of-the-art on various benchmarks. To further validate our performance and facilitate future research in text-rich canvas layout generation, we also construct a challenging text-rich dataset named TRich500, which contains a wide variety of pre-existing text images from the real-world.

Integrated sensing and communication (ISAC) has emerged as a transformative paradigm, merging the capabilities of sensing and communication to enhance efficiency and enable advanced applications. Mobile crowdsensing (MCS), as a important example of ISAC, leverages unmanned vehicles such as UAVs to continuously gather and transmit environmental data, supporting critical applications like traffic monitoring, urban congestion management, and accident investigation. In this paper, we focus on multi-task-oriented UAV crowdsensing (UCS), where diverse tasks—such as surveillance and emergency response—each have distinct age-of-information (AoI) requirements. We introduce a novel metric, the “valid task handling index,” to evaluate the performance of handling multiple tasks effectively. Our proposed hierarchical multi-agent deep reinforcement learning (MADRL) framework, DRL-MTUCS, integrates seamlessly with multi-agent actor-critic reinforcement learning methods. It features dynamically weighted queues for UAV goal assignment, enabling efficient management of multiple emergency tasks, and a low-level UAV execution module with a self-balancing intrinsic reward mechanism. This ensures all tasks are completed within their individual AoI constraints. Extensive experiments and trajectory visualizations validate the superior performance and robustness of DRL-MTUCS compared to six baselines across varying conditions, including the number of UAVs, surveillance task AoI thresholds, and emergency task image blur requirements.

Large vision and language models deployed at edge encounter continuously evolving input distributions, including not only new tasks but also highly unbalanced long-tail classes. For example, smart-surveillance cameras frequently capture common objects such as pedestrians and cars, while only occasionally observing tail classes like horseback riders or stroller pushers. However, most existing long-tail mitigation techniques are designed for fixed pretraining data. This often leads to poor accuracy of tail classes on evolving data and incurs high computational costs on edge devices. In this article, we propose EdgeTail, a lightweight long-tail mitigation method for edge-side continual learning. EdgeTail’s key design features are: (i) optimal long-tailed mitigation solution search, which adaptively selects the best long-tail learning method for the current distribution/task; and (ii) graph attention classifier and multi-branch adapter, which improves the quality and stability of tail class representations with small overheads. We implement EdgeTail in PyTorch and extensively evaluate it against state-of-the-art methods. The results show that EdgeTail improves the average accuracy by 36.09% under fixed training windows and by 31.12% under different training window sizes.

Indoor localization is crucial for applications such as navigation, asset tracking, and emergency response. Fingerprint-based methods that use RSSI are widely adopted; however, they fail under large environmental changes. Unmanned Vehicles (UVs) equipped with high precision sensors are able to collect fingerprints, serving as a promising way by forming a Vehicular Crowdsensing (VCS) campaign. In this paper, we propose “BRAVE”, a Bayesian RL Approach for VCS under Environment changing, while introducing a new metric “Calibration Benefit” to explicitly quantify how effectively a learned trajectory updates those regions of the fingerprint database that have changed and matter most for localization. Specifically, we propose a spatial-temporal Bayesian Network(BN) for change detection, a region rearrangement method for fewer restarts, and an optimistic strategy to balance the exploration and exploitation trade-offs in optimizing calibration benefit. Extensive results on two real-world datasets from SML Center (Shanghai) and Haopu Fashion City (Shanghai) demonstrate that BRAVE outperforms eight baselines and the derived dataset has better localization accuracy compared with the original dataset.

Vehicles are transforming from transportation means to mobile intelligent terminals, powered by artificial intelligence (AI) technologies such as large language (LLM) and vision large models (VLM). To maintain high learning accuracy in all situations, in-vehicle AI models encounter the acute need to continuously learn and dynamically retrain AI models on the fly. The volatile resource demands of such stochastic retraining jobs and inference jobs of higher priority however can not be accommodated simultaneously by existing training systems, which rely on pre-generated compressed models of fixed architecture. In this paper, we proposeConvertNet, a novel continuous learning system that effectively scales up/down a compressed model through training-time neuron memorizer (TNM) - a max heap of neurons in tree structure. Based on TNM,ConvertNetestimates the model's resource-accuracy trade-off per neuron, thus efficiently utilizing the limited available resources to scale up and (re)train the model to improve its accuracy. Evaluated on six representative in-vehicle scenarios, comparative experiments against eleven state-of-the-art techniques show thatConvertNetachieves as much as 22.33% improvement in learning accuracy, and reduces energy consumption by 3.65x.

Team-based combat scenarios are prevalent in various real-world applications like video gaming. Analyzing tactics in these scenarios is essential for gaining insights into game processes and improving combat behaviors. The decision-making data in team-based combat include character actions, movement trajectories, and event sequences. Existing studies face challenges in visualizing and analyzing combat tactics due to the complexity and the multifaceted characteristics of the decision-making data. To address these challenges, we introduce TactiVis, a visual analytics system designed for analyzing combat decision-making behavior. Using MOBA game as a representative case of team-based combat, TactiVis adopts a macro-to-micro tactics visual analytics framework consisting of three stages: match-level analysis, event-level understanding, and character-level comparison. In the TactiVis system, we introduce the v-storyline visualization, which encodes positions along the vertical axis to reveal tactical patterns. Case studies and a usability study demonstrate the utility and usability of TactiVis for helping analysts understand combat patterns and analyze tactics.

Effective Class Incremental Segmentation (CIS) requires simultaneously mitigating catastrophic forgetting and ensuring sufficient plasticity to integrate new classes. The inherent conflict above often leads to a back-and-forth, which turns the objective into finding the balance between the performance of previous (old) and incremental (new) classes. To address this conflict, we introduce a novel approach, Conflict Mitigation via Branched Optimization (CoMBO). Within this approach, we present the Query Conflict Reduction module, designed to explicitly refine queries for new classes through lightweight, class-specific adapters. This module provides an additional branch for the acquisition of new classes while preserving the original queries for distillation. Moreover, we develop two strategies to further mitigate the conflict following the branched structure, i.e., the Half-Learning Half-Distillation (HDHL) over classification probabilities, and the Importance-Based Knowledge Distillation (IKD) over query features. HDHL selectively engages in learning for classification probabilities of queries that match the ground truth of new classes, while aligning unmatched ones to the corresponding old probabilities, thus ensuring retention of old knowledge while absorbing new classes via learning negative samples. Meanwhile, IKD assesses the importance of queries based on their matching degree to old classes, prioritizing the distillation of important features and allowing less critical features to evolve. Extensive experiments in Class Incremental Panoptic and Semantic Segmentation settings have demonstrated the superior performance of CoMBO. Project page: https://guangyu-ryan.github.io/CoMBO.

Recent advances in vision-language models (VLMs) have significantly improved performance in embodied tasks such as goal decomposition and visual comprehension. However, providing accurate rewards for robotic manipulation without fine-tuning VLMs remains challenging due to the absence of domain-specific robotic knowledge in pre-trained datasets and high computational costs that hinder real-time applicability. To address this, we propose $\mathrm{T}^2$-VLM, a novel training-free, temporally consistent framework that generates accurate rewards through tracking the status changes in VLM-derived subgoals. Specifically, our method first queries the VLM to establish spatially aware subgoals and an initial completion estimate before each round of interaction. We then employ a Bayesian tracking algorithm to update the goal completion status dynamically, using subgoal hidden states to generate structured rewards for reinforcement learning (RL) agents. This approach enhances long-horizon decision-making and improves failure recovery capabilities with RL. Extensive experiments indicate that $\mathrm{T}^2$-VLM achieves state-of-the-art performance in two robot manipulation benchmarks, demonstrating superior reward accuracy with reduced computation consumption. We believe our approach not only advances reward generation techniques but also contributes to the broader field of embodied AI. Project website: https://t2-vlm.github.io/.

Executing transformer-based applications on edge devices encounters challenging scenarios of continually learning new tasks. Federated continual learning (FCL) is a prevalent framework that supports model training using local data across edge devices. However, state-of-the-art FCL techniques either cause high computation and communication costs on large transformer models, or train small/compressed models, limiting both the learning capacity/accuracy on clients' local data and the global knowledge exchange among them. In this paper, we propose ElasticFed, a neuron-grained scaling approach for large-small transformer collaborative training in edge-based FCL. ElasticFed's key design features are (i) a proxy mechanism in local training, which constructs a compact model consisting of the original transformer's most important neurons to the current task, thus collaboratively training both models with small overheads; and (ii) a neuron-grained global aggregator, which se-lectively aggregates different clients' knowledge belonging to the most important neurons, thus maximizing the positive knowledge transfer with small communication costs. The comparison results against state-of-the-art techniques show that ElasticFed improves accuracy by 59.69% under the same training time. Compared to the techniques on original transformers, ElasticFed reduces training time and communication costs by 1.5x and 2.9x with small accuracy losses of 0.88%

Teleoperation is essential for autonomous robot learning, especially in manipulation tasks that require human demonstrations or corrections. However, most existing systems only offer unilateral robot control and lack the ability to synchronize the robot’s status with the teleoperation hardware, preventing real-time, flexible intervention. In this work, we introduce HACTS (Human-As-Copilot Teleoperation System), a novel system that establishes bilateral, real-time joint synchronization between a robot arm and teleoperation hardware. This simple yet effective feedback mechanism, akin to a steering wheel in autonomous vehicles, enables the human copilot to intervene seamlessly while collecting action-correction data for future learning. Implemented using 3D-printed components and low-cost, off-the-shelf motors, HACTS is both accessible and scalable. Our experiments show that HACTS significantly enhances performance in imitation learning (IL) and reinforcement learning (RL) tasks, boosting IL recovery capabilities and data efficiency, and facilitating human-in-the-loop RL. HACTS paves the way for more effective and interactive human-robot collaboration and data-collection, advancing the capabilities of robot manipulation.

Insights in tabular data capture valuable patterns that help analysts understand critical information.Organizing related insights into visual data stories is crucial for in-depth analysis.However, constructing such stories is challenging because of the complexity of the inherent relations between extracted insights.Users face difficulty sifting through a vast number of discrete insights to integrate specific ones into a unified narrative that meets their analytical goals.Existing methods either heavily rely on user expertise, making the process inefficient, or employ automated approaches that cannot fully capture their evolving goals.In this paper, we introduce InRe-AcTable, a framework that enhances visual data story construction by establishing both structural and semantic connections between data insights.Each user interaction triggers the Acting module, which utilizes an insight graph for structural filtering to narrow the search space, followed by the Reasoning module using the retrievalaugmented generation method based on large language models for semantic filtering, ultimately providing insight recommendations aligned with the user's analytical intent.Based on the InReAcTable framework, we develop an interactive prototype system that guides users to construct visual data stories aligned with their analytical requirements.We conducted a case study and a user experiment to demonstrate the utility and effectiveness of the InReAcTable framework and the prototype system for interactively building visual data stories.

Media bias refers to bias in news reporting and coverage that exists pervasively. By identifying media bias, social scientists can understand the different perspectives held by media outlets in news reporting. Existing studies only focus on the analysis of media bias of isolated incidents, but neglect their sustained characteristics. Thus, they cannot provide a comprehensive understanding of specific news topics. We develop BiaSeer, a visual analytics system for identifying and understanding sustained bias of media outlets. BiaSeer employs an overview-to-detail approach for interactive identification of media bias. The overview assists users in determining the analysis scope of media outlets. In addition, it visualizes the variances in coverage patterns between selected media outlets using a matrix visualization to facilitate the identification of biased news articles. BiaSeer visualizes the sustained bias in the context of the evolution of events. It first summarizes news articles into events based on a keyword co-occurrence graph and then connects events into a narrative structure using a path-aware story tree construction method. In addition, BiaSeer integrates a sustained bias computation algorithm and enables analysts to compare the narrative structures of different media outlets using the juxtaposition-based visualization approach. We conducted a user experiment to validate the effectiveness of BiaSeer in helping social scientists understand news topics and the usability of visualization designs. To examine the effectiveness of BiaSeer, we conducted a case study with social scientists on the topics of the Russia-Ukraine conflict. The results demonstrate the utility and usability of BiaSeer in efficiently analyzing media bias and attaining a well-rounded understanding of news topics.

Few-shot Semantic Segmentation (FSS) aims to accurately segment query images with guidance from only a few annotated support images. Previous methods typically rely on pixel-level feature correlations, denoted as the many-to-many (pixels-to-pixels) or few-to-many (prototype-to-pixels) manners. Recent mask proposals classification pipeline in semantic segmentation enables more efficient few-to-few (prototype-to-prototype) correlation between masks of query proposals and support reference. However, these methods still involve intermediate pixel-level feature correlation, resulting in lower efficiency. In this paper, we introduce the Proposal and Reference masks matching transFormer (PRFormer), designed to rigorously address mask matching in both spatial and semantic aspects in a thorough few-to-few manner. Following the mask-classification paradigm, PRFormer starts with a class-agnostic proposal generator to partition the query image into proposal masks. It then evaluates the features corresponding to query proposal masks and support reference masks using two strategies: semantic matching based on feature similarity across prototypes and spatial matching through mask intersection ratio. These strategies are implemented as the Prototype Contrastive Correlation (PrCC) and Prior-Proposals Intersection (PPI) modules, respectively. These strategies enhance matching precision and efficiency while eliminating dependence on pixel-level feature correlations. Additionally, we propose the category discrimination NCE (cdNCE) loss and IoU-KLD loss to constrain the adapted prototypes and align the similarity vector with the corresponding IoU between proposals and ground truth. Given that class-agnostic proposals tend to be more accurate for training classes than for novel classes in FSS, we introduce the Weighted Proposal Refinement (WPR) to refine the most confident masks with detailed features, yielding more precise predictions. Experiments on the popular Pascal-5i and COCO-20i benchmarks show that our Few-to-Few approach, PRFormer, outperforms previous methods, achieving mIoU scores of 70.4% and 49.4%, respectively, on 1-shot segmentation. Code is available athttps://github.com/ANDYZAQ/PRFormer.

By harnessing the capabilities of unmanned aerial and ground vehicles (UAVs and UGVs), equipped with high-precision sensors, air-ground mobile crowdsensing (AG-MCS) has proven to be effective for data collection in urban environments. In this paper, by optimizing the metric of age-of-information (AoI) that measures the freshness of collected data, we consider the problem of AoI-Aware AG-MCS (A3G-MCS), where UGVs dispatch UAVs from multiple UGV stops to collect data from point-of-interests (PoIs). We propose a novel multi-agent curriculum learning framework called “MACL(MCS)”, that explicitly balances the individual and team goals of both UAV/UGV controllers to facilitate the exploration of policy towards globally-optimal performance. It is further enhanced by a UAV/UGV collaborative observation augmentation (COA) module for improved inter-controller communication. Extensive results reveal that MACL(MCS) consistently outperforms five baselines, and achieves comparable performance to exact method with better scalability and efficiency. It also showcases strong generalization capability towards real-world scenarios on both TSPLIB and Purdue, KAIST and NCSU datasets.

Foundation models (FMs) such as large language models are becoming the backbone technology for artificial intelligence systems. It is particularly challenging to deploy multiple FMs on edge devices, which not only have limited computational resources, but also encounter unseen input data from evolving domains or learning tasks. When new data arrives, existing prior art of FM mainly focuses on retraining compressed models of predetermined network architectures, limiting the feasibility of edge devices to efficiently achieve high accuracy for FMs. In this paper, we propose EdgeTA, a neuron-grained FM scaling system to maximize the overall accuracy of FMs promptly in response to their data dynamics. EdgeTA's key design features in scaling are (i) proxy mechanism, which adaptively transforms a FM into a compact architecture retaining the most important neurons to the input data, and (ii) neuron-grained scheduler, which jointly optimizes model sizes and resource allocation for all FMs on edge devices. Under tight retraining window and limited device resources, the design of EdgeTA can achieve most of the original FM's accuracy with much smaller retraining costs. We implement EdgeTA on FMs of natural language processing, computer vision and multimodal applications. Comparison results against state-of-the-art techniques show that our approach improves accuracy by 21.88% and reduces memory footprint and energy consumptions by 27.14% and 65.65%, while further achieving 15.96% overall accuracy improvement via neuron-grained scheduling.

Offline reinforcement learning (RL), which operates solely on static datasets without further interactions with the environment, provides an appealing alternative to learning a safe and promising control policy. The prevailing methods typically learn a conservative policy to mitigate the problem of Q-value overestimation, but it is prone to overdo it, leading to an overly conservative policy. Moreover, they optimize all samples equally with fixed constraints, lacking the nuanced ability to control conservative levels in a fine-grained manner. Consequently, this limitation results in a performance decline. To address the above two challenges in a united way, we propose a framework, adaptive conservative level in Q-learning (ACL-QL), which limits the Q-values in a mild range and enables adaptive control on the conservative level over each state-action pair, i.e., lifting the Q-values more for good transitions and less for bad transitions. We theoretically analyze the conditions under which the conservative level of the learned Q-function can be limited in a mild range and how to optimize each transition adaptively. Motivated by the theoretical analysis, we propose a novel algorithm, ACL-QL, which uses two learnable adaptive weight functions to control the conservative level over each transition. Subsequently, we design a monotonicity loss and surrogate losses to train the adaptive weight functions, Q-function, and policy network alternatively. We evaluate ACL-QL on the commonly used datasets for deep data-driven reinforcement learning (D4RL) benchmark and conduct extensive ablation studies to illustrate the effectiveness and state-of-the-art performance compared with existing offline DRL baselines.

Heterogeneous domain adaptation (HDA) is expected to achieve effective knowledge transfer from a label-rich source domain to a heterogeneous target domain with scarce labeled data. Most prior HDA methods strive to align the cross-domain feature distributions by learning domain invariant representations without considering the intrinsic semantic correlations among categories, which inevitably results in the suboptimal adaptation performance across domains. Therefore, to address this issue, we propose a novel semantic correlation transfer (SCT) method for HDA, which not only matches the marginal and conditional distributions between domains to mitigate the large domain discrepancy, but also transfers the category correlation knowledge underlying the source domain to target by maximizing the pairwise class similarity across source and target. Technically, the domainwise and classwise centroids (prototypes) are first computed and aligned according to the feature embeddings. Then, based on the derived classwise prototypes, we leverage the cosine similarity of each two classes in both domains to transfer the supervised source semantic correlation knowledge among different categories to target effectively. As a result, the feature transferability and category discriminability can be simultaneously improved during the adaptation process. Comprehensive experiments and ablation studies on standard HDA tasks, such as text-to-image, image-to-image, and text-to-text, have demonstrated the superiority of our proposed SCT against several state-of-the-art HDA methods.

Federated continual learning (FCL) has attracted growing attention in achieving collaborative model training among edge clients, each of which learns its local model for a sequence of tasks. Most existing FCL approaches aggregate clients’ latest local models to exchange knowledge. This unfortunately deviates from real-world scenarios where each model is optimized independently using the client’s own dynamic data and different clients have heterogeneous tasks. These tasks not only have distinct class labels (e.g., animals or vehicles) but also differ in input feature distributions. The aggregated model thus often shifts to a higher loss value and incurs accuracy degradation. In this article, we depart from the model-grained view of aggregation and transform it into multiple task-grained aggregations. Each aggregation allows a client to learn from other clients to improve its model accuracy on one task. To this end, we propose Loci to provide abstractions for clients’ past and peer task knowledge using compact model weights, and develop a communication-efficient approach to train each client’s local model by exchanging its tasks’ knowledge with the most accuracy relevant one from other clients. Through its general-purpose API, Loci can be used to provide efficient on-device training for existing deep learning applications of graph, image, nature language processing, and multimodal data. Using extensive comparative evaluations, we show Loci improves the model accuracy by 32.48% without increasing training time, reduces communication cost by 83.6%, and achieves more improvements when scale (task/client number) increases.

Embedding visual representations within original hierarchical tables can mitigate additional cognitive load stemming from the division of users' attention. The created hierarchical table visualizations can help users understand and explore complex data with multi-level attributes. However, because of many options available for transforming hierarchical tables and selecting subsets for embedding, the design space of hierarchical table visualizations becomes vast, and the construction process turns out to be tedious, hindering users from constructing hierarchical table visualizations with many data insights efficiently. We propose InsigHTable, a mixed-initiative and insight-driven hierarchical table transformation and visualization system. We first define data insights within hierarchical tables, which consider the hierarchical structure in the table headers. Since hierarchical table visualization construction is a sequential decision-making process, InsigHTable integrates a deep reinforcement learning framework incorporating an auxiliary rewards mechanism. This mechanism addresses the challenge of sparse rewards in constructing hierarchical table visualizations. Within the deep reinforcement learning framework, the agent continuously optimizes its decision-making process to create hierarchical table visualizations to uncover more insights by collaborating with analysts. We demonstrate the usability and effectiveness of InsigHTable through two case studies and sets of experiments. The results validate the effectiveness of the deep reinforcement learning framework and show that InsigHTable can facilitate users to construct hierarchical table visualizations and understand underlying data insights.

When using exploratory visual analysis to examine multivariate hierarchical data, users often need to query data to narrow down the scope of analysis. However, formulating effective query expressions remains a challenge for multivariate hierarchical data, particularly when datasets become very large. To address this issue, we develop a declarative grammar, HiRegEx (Hierarchical data Regular Expression), for querying and exploring multivariate hierarchical data. Rooted in the extended multi-level task topology framework for tree visualizations (e-MLTT), HiRegEx delineates three query targets (node, path, and subtree) and two aspects for querying these targets (features and positions), and uses operators developed based on classical regular expressions for query construction. Based on the HiRegEx grammar, we develop an exploratory framework for querying and exploring multivariate hierarchical data and integrate it into the TreeQueryER prototype system. The exploratory framework includes three major components: top-down pattern specification, bottom-up data-driven inquiry, and context-creation data overview. We validate the expressiveness of HiRegEx with the tasks from the e-MLTT framework and showcase the utility and effectiveness of TreeQueryER system through a case study involving expert users in the analysis of a citation tree dataset.

Enabling deep models to generalize in non-stationary environments is vital for real-world machine learning, as data distributions are often found to continually change. Recently, evolving domain generalization (EDG) has emerged to tackle the domain generalization in a time-varying system, where the domain gradually evolves over time in an underlying continuous structure. Nevertheless, it typically assumes multiple source domains simultaneously ready. It still remains an open problem to address EDG in the domain-incremental setting, where source domains are non-static and arrive sequentially to mimic the evolution of training domains. To this end, we propose Weight Diffusion (W-Diff), a novel framework that utilizes the conditional diffusion model in the parameter space to learn the evolving pattern of classifiers during the domain-incremental training process. Specifically, the diffusion model is conditioned on the classifier weights of different historical domain (regarded as a reference point) and the prototypes of current domain, to learn the evolution from the reference point to the classifier weights of current domain (regarded as the anchor point). In addition, a domain-shared feature encoder is learned by enforcing prediction consistency among multiple classifiers, so as to mitigate the overfitting problem and restrict the evolving pattern to be reflected in the classifier as much as possible. During inference, we adopt the ensemble manner based on a great number of target domain-customized classifiers, which are cheaply obtained via the conditional diffusion model, for robust prediction. Comprehensive experiments on both synthetic and real-world datasets show the superior generalization performance of W-Diff on unseen domains in the future.

Indoor localization is drawing more and more attentions due to the growing demand of various location-based services, where fingerprinting is a popular data driven techniques that does not rely on complex measurement equipment, yet it requires site surveys which is both labor-intensive and time-consuming. Vehicular crowdsensing (VCS) with unmanned vehicles (UVs) is a novel paradigm to navigate a group of UVs to collect sensory data from certain point-of-interests periodically (PoIs, i.e., coverage holes in localization scenarios). In this paper, we formulate the multi-floor indoor fingerprint collection task with periodical PoI coverage requirements as a constrained optimization problem. Then, we propose a multi-agent deep reinforcement learning (MADRL) based solution, “MADRL-PosVCS”, which consists of a primal-dual framework to transform the above optimization problem into the unconstrained duality, with adjustable Lagrangian multipliers to ensure periodic fingerprint collection. We also propose a novel intrinsic reward mechanism consists of the mutual information between a UV’s observations and environment transition probability parameterized by a Bayesian Neural Network (BNN) for exploration, and a elevator-based reward to allow UVs to go cross different floors for collaborative fingerprint collections. Extensive simulation results on three real-world datasets in SML Center (Shanghai), Joy City (Hangzhou) and Haopu Fashion City (Shanghai) show that MADRL-PosVCS achieves better results over four baselines on fingerprint collection ratio, PoI coverage ratio for collection intervals, geographic fairness and average moving distance.

Metaverse is expected to provide mobile users with emerging applications both in regular situation like intelligent transportation services and in emergencies like wireless search and disaster response. These applications are usually associated with stringent quality-of-information (QoI) requirements like throughput and age-of-information (AoI), which can be further guaranteed by using unmanned aerial vehicles (UAVs) as aerial base stations (BSs) to compensate the existing 5G infrastructures. In this paper, we consider a new QoI-aware mobile crowdsensing (MCS) campaign by UAVs which move around and collect data from mobile users wearing metaverse devices. Specifically, we propose “MetaCS”, a multi-agent deep reinforcement learning (MADRL) framework with improvements on a Transformer-based user mobility prediction module between regions and a relational graph learning mechanism to enable the selection of most informative partners to communicate for each UAV. Extensive results and trajectory visualizations on three real mobility datasets in NCSU, KAIST and Beijing show that MetaCS consistently outperforms six baselines in terms of overall QoI index, when varying different numbers of UAVs, throughput requirement, and AoI threshold.

The integrated ground-air-space (GAS) communications system can enhance post-disaster rescue and management efforts when traditional networks fail, by navigating unmanned ground vehicles (UGVs) and unmanned arieal vehicles (UAVs) to collaboratively collect sufficient data from point-of-interests (PoIs) in a timely manner. In this paper, we consider the GAS vehicular crowdsensing (VCS) campaign, where UGVs dispatch and callback UAVs periodically across multiple stops in the workzone, to maximize the total collected amount of data, geographic fairness while minimizing the energy consumption simultaneously. Specifically, we propose an energy-efficient, go-directed hierarchical multi-agent deep reinforcement learning (MADRL) method with discrete diffusion models called “gMADRL-VCS”, to optimize the high-level goal-conditioned navigation policies of UGVs, and the low-level long-term sensing strategies of UAVs. Extensive experimental results on two real-world datasets in Roma, Italy, and Hong Kong SAR, China show that gMADRL-VCS outperforms baselines in terms of energy efficiency, data collection ratio, energy consumption, and UAV-UGV cooperation factor.

Online display advertising platforms service numerous advertisers by providing real-time bidding (RTB) for the scale of billions of ad requests every day. The bidding strategy handles ad requests cross multiple channels to maximize the number of clicks under the set financial constraints, i.e., total budget and cost-per-click (CPC), etc. Different from existing works mainly focusing on single channel bidding, we explicitly consider cross-channel constrained bidding with budget allocation. Specifically, we propose a hierarchical offline deep reinforcement learning (DRL) framework called “HiBid”, consisted of a high-level planner equipped with auxiliary loss for non-competitive budget allocation, and a data augmentation enhanced low-level executor for adaptive bidding strategy in response to allocated budgets. Additionally, a CPC-guided action selection mechanism is introduced to satisfy the cross-channel CPC constraint. Through extensive experiments on both the large-scale log data and online A/B testing, we confirm that HiBid outperforms six baselines in terms of the number of clicks, CPC satisfactory ratio, and return-on-investment (ROI). We also deploy HiBid on Meituan advertising platform to already service tens of thousands of advertisers every day.

Executing deep neural networks (DNN) based vision tasks on edge devices encounters challenging scenarios of significant and continually evolving data domains (e.g. background or subpopulation shift). With limited resources, the state-of-the-art domain adaptation (DA) methods either cause high training overheads on large DNN models, or incur significant accuracy losses when adapting small/compressed models in an online fashion. The inefficient resource scheduling among multiple applications further degrades their overall model accuracy. In this paper, we present ElasticDNN, a framework that enables online DNN remodeling for applications encountering evolving domain drifts at edge. Its first key component is the master-surrogate DNN models, which can dynamically generate a small surrogate DNN by retaining and training the large master DNN's most relevant regions pertinent to the new domain. The second novelty of ElasticDNN is the filter-grained resource scheduling, which allocates GPU resources based on online accuracy estimation and DNN remodeling of co-running applications. We fully implement ElasticDNN and demonstrate its effectiveness through extensive experiments. The results show that, compared to existing online DA methods using the same model sizes, ElasticDNN improves accuracy by 23.31% and reduces adaption time by 35.67x. In the more challenging multi-application scenario, ElasticDNN improves accuracy by an average of 25.91%.

Unmanned aerial vehicle (UAV) crowdsensing (UCS) is an emerging data collection paradigm to provide reliable and high quality urban sensing services, with age-of-information (AoI) requirement to measure data freshness in real-time applications. In this paper, we explicitly consider the case to ensure that the attained AoI always stay within a specific threshold. The goal is to maximize the total amount of collected data from diverse Point-of-Interests (PoIs) while minimizing AoI and AoI threshold violation ratio under limited energy supplement. To this end, we propose a decentralized multi-agent deep reinforcement learning framework called “DRL-UCS($\text {AoI}_{th}$)” for multi-UAV trajectory planning, which consists of a novel transformer-enhanced distributed architecture and an adaptive intrinsic reward mechanism for spatial cooperation and exploration. Extensive results and trajectory visualization on two real-world datasets in Beijing and San Francisco show that, DRL-UCS($\text {AoI}_{th}$) consistently outperforms all nine baselines when varying the number of UAVs, AoI threshold and generated data amount in a timeslot.

Extracting data insights and generating visual data stories from tabular data are critical parts of data analysis. However, most existing studies primarily focus on tabular data stored as flat tables, typically without leveraging the relations between cells in the headers of hierarchical tables. When properly used, rich table headers can enable the extraction of many additional data stories. To assist analysts in visual data storytelling, an approach is needed to organize these data insights efficiently. In this work, we propose CoInsight, a system to facilitate visual storytelling for hierarchical tables by connecting insights. CoInsight extracts data insights from hierarchical tables and builds insight relations according to the structure of table headers. It further visualizes related data insights using a nested graph with edge bundling. We evaluate the CoInsight system through a usage scenario and a user experiment. The results demonstrate the utility and usability of CoInsight for converting data insights in hierarchical tables into visual data stories.

Generalizing models trained on normal visual conditions to target domains under adverse conditions is demanding in the practical systems. One prevalent solution is to bridge the domain gap between clear- and adverse-condition images to make satisfactory prediction on the target. However, previous methods often reckon on additional reference images of the same scenes taken from normal conditions, which are quite tough to collect in reality. Furthermore, most of them mainly focus on individual adverse condition such as nighttime or foggy, weakening the model versatility when encountering other adverse weathers. To overcome the above limitations, we propose a novel framework, Visibility Boosting and Logit-Constraint learning (VBLC), tailored for superior normal-toadverse adaptation. VBLC explores the potential of getting rid of reference images and resolving the mixture of adverse conditions simultaneously. In detail, we first propose the visibility boost module to dynamically improve target images via certain priors in the image level. Then, we figure out the overconfident drawback in the conventional cross-entropy loss for self-training method and devise the logit-constraint learning, which enforces a constraint on logit outputs during training to mitigate this pain point. To the best of our knowledge, this is a new perspective for tackling such a challenging task. Extensive experiments on two normal-to-adverse domain adaptation benchmarks, i.e., Cityscapes to ACDC and Cityscapes to FoggyCityscapes + RainCityscapes, verify the effectiveness of VBLC, where it establishes the new state of the art. Code is available at https://github.com/BIT-DA/VBLC.

Explicit visible videos can provide sufficient visual information and facilitate vision applications. Unfortunately, the image sensors of visible cameras are sensitive to light conditions like darkness or overexposure. To make up for this, recently, infrared sensors capable of stable imaging have received increasing attention in autonomous driving and monitoring. However, most prosperous vision models are still trained on massive clear visible data, facing huge visual gaps when deploying to infrared imaging scenarios. In such cases, transferring the infrared video to a distinct visible one with fine-grained semantic patterns is a worthwhile endeavor. Previous works improve the outputs by equally optimizing each patch on the translated visible results, which is unfair for enhancing the details on content-rich patches due to the long-tail effect of pixel distribution. Here we propose a novel CPTrans framework to tackle the challenge via balancing gradients of different patches, achieving the fine-grained Content-rich Patches Transferring. Specifically, the content-aware optimization module encourages model optimization along gradients of target patches, ensuring the improvement of visual details. Additionally, the content-aware temporal normalization module enforces the generator to be robust to the motions of target patches. Moreover, we extend the existing dataset InfraredCity to more challenging adverse weather conditions (rain and snow), dubbed as InfraredCity-Adverse11The code and dataset are available at https://github.com/BIT-DA/12V-Processing, Extensive experiments show that the proposed CPTrans achieves state-of-the-art performance under diverse scenes while requiring less training time than competitive methods.

The development of vision models for real-world applications is hindered by the challenge of annotated data scarcity, which has necessitated the adoption of dataefficient visual learning techniques such as semi-supervised learning. Unfortunately, the prevalent cross-entropy supervision is limited by its focus on category discrimination while disregarding the semantic connection between concepts, which ultimately results in the suboptimal exploitation of scarce labeled data. To address this issue, this paper presents a novel approach that seeks to leverage linguistic knowledge for data-efficient visual learning. The proposed approach, BorLan, Borrows knowledge from off-theshelf pretrained Language models that are already endowed with rich semantics extracted from large corpora, to compensate the semantic deficiency due to limited annotation in visual training. Specifically, we design a distribution alignment objective, which guides the vision model to learn both semantic-aware and domain-agnostic representations for the task through linguistic knowledge. One significant advantage of this paradigm is its flexibility in combining various visual and linguistic models. Extensive experiments on semi-supervised learning, single domain generalization and few-shot learning validate its effectiveness. Code is available at https://github.com/BIT-DA/BorLan.

Class incremental semantic segmentation aims to strike a balance between the model’s stability and plasticity by maintaining old knowledge while adapting to new concepts. However, most state-of-the-art methods use the freeze strategy for stability, which compromises the model’s plasticity. In contrast, releasing parameter training for plasticity could lead to the best performance for all categories, but this requires discriminative feature representation. Therefore, we prioritize the model’s plasticity and propose the Contrast inter- and intra-class representations for Incremental Segmentation (CoinSeg), which pursues discriminative representations for flexible parameter tuning. Inspired by the Gaussian mixture model that samples from a mixture of Gaussian distributions, CoinSeg emphasizes intra-class diversity with multiple contrastive representation centroids. Specifically, we use mask proposals to identify regions with strong objectness that are likely to be diverse instances/centroids of a category. These mask proposals are then used for contrastive representations to reinforce intra-class diversity. Meanwhile, to avoid bias from intra-class diversity, we also apply category-level pseudo-labels to enhance category-level consistency and inter-category diversity. Additionally, CoinSeg ensures the model’s stability and alleviates forgetting through a specific flexible tuning strategy. We validate CoinSeg on Pascal VOC 2012 and ADE20K datasets with multiple incremental scenarios and achieve superior results compared to previous state-of-the-art methods, especially in more challenging and realistic long-term scenarios. Code is available at https://github.com/zkzhang98/CoinSeg.

Spatial Crowdsourcing (SC) has been proved as an effective paradigm for data acquisition in urban environments. Apart from using human participants, with the rapid development of unmanned vehicles (UVs) technologies, unmanned aerial or ground vehicles (UAVs, UGVs) are equipped with various high-precision sensors, enabling them to become new types of data collectors. However, UGVs’ operational range is constrained by the road network, and UAVs are limited by power supply, it is thus natural to use UGVs and UAVs together as a coalition, and more precisely, UGVs behave as the UAV carriers for range extensions to achieve complicated air-ground SC tasks. In this paper, we propose a novel communication-based multi-agent deep reinforcement learning method called "GARL", which consists of a multi-center attention-based graph convolutional network (GCN) to accurately extract UGV specific features from UGV stop network called "MC-GCN", and a novel GNN-based communication mechanism called "E-Comm" to make the cooperation among UGVs adaptive to constant changing of geometric shapes formed by UGVs. Extensive simulation results on two campuses of KAIST and UCLA campuses show that GARL consistently outperforms eight other baselines in terms of overall efficiency.

Personalized location prediction is key to many mobile applications and services. In this paper, motivated by both statistical and visualized preliminary analysis on three real datasets, we observe a strong spatiotemporal correlation for user trajectories among the visited area-of-interests (AoIs) and different time periods on both weekly and daily basis, which directly motivates our time-aware location prediction model design called "t-LocPred". It models the spatial correlations among AoIs by coarse-grained convolutional processing of the user trajectories in AoIs of different time periods ("ConvAoI"); and predicts his/her fine-grained next visited PoI using a novel memory-augmented attentive LSTM model ("mem-attLSTM") to capture long-term behavior patterns. Experimental results show that t-LocPred outperforms 8 baselines. We also show the impact of hyperparameters and the benefits ConvAoI can bring to these baselines.

Deep Neural Networks (DNNs) have been ubiquitously adopted in internet of things and are becoming an integral of our daily life. When tackling the evolving learning tasks in real world, such as classifying different types of objects, DNNs face the challenge to continually retrain themselves according to the tasks on different edge devices. Federated continual learning is a promising technique that offers partial solutions but yet to overcome the following difficulties: the significant accuracy loss due to the limited on-device processing, the negative knowledge transfer caused by the limited communication of non-IID data, and the limited scalability on the tasks and edge devices. In this paper, we propose FedKNOW, an accurate and scalable federated continual learning framework, via a novel concept of signature task knowledge. FedKNOW is a client side solution that continuously extracts and integrates the knowledge of signature tasks which are highly influenced by the current task. Each client of FedKNOW is composed of a knowledge extractor, a gradient restorer and, most importantly, a gradient integrator. Upon training for a new task, the gradient integrator ensures the prevention of catastrophic forgetting and mitigation of negative knowledge transfer by effectively combining signature tasks identified from the past local tasks and other clients’ current tasks through the global model. We implement FedKNOW in PyTorch and extensively evaluate it against state-of-the-art techniques using popular federated continual learning benchmarks. Extensive evaluation results on heterogeneous edge devices show that FedKNOW improves model accuracy by 63.24% without increasing model training time, reduces communication cost by 34.28%, and achieves more improvements under difficult scenarios such as large numbers of tasks or clients, and training different complex networks.

Spatial crowdsourcing (SC) has proven as a promising paradigm to employ human workers to collect data from diverse Point-of-Interests (PoIs) in a given area. Different from using human participants, we propose a novel air-ground SC scenario to fully take advantage of benefits brought by unmanned vehicles (UVs), including unmanned aerial vehicles (UAVs) with controllable high mobility and unmanned ground vehicles (UGVs) with abundant sensing resources. The objective is to maximize the amount of collected data, geographical fairness among all PoIs, and minimize the data loss and energy consumption, integrated as one single metric called "efficiency". We explicitly explore both individuality and cooperation natures of UAVs and UGVs by proposing a multi-agent deep reinforcement learning (MADRL) framework called "h/i-MADRL". Compatible with all multi-agent actor-critic methods, h/i-MADRL adds two novel plug-in modules: (a) h-CoPO, which models the cooperation preference among heterogeneous UAVs and UGVs; and (b) i-EOI, which extracts the UV’s individuality and encourages a better spatial division of work by adding intrinsic reward. Extensive experimental results on two real-world datasets on Purdue and NCSU campuses confirm that h/i-MADRL achieves a better exploration of both individuality and cooperation simultaneously, resulting in a better performance in terms of efficiency compared with five baselines.

Vision applications powered by deep neural networks (DNNs) are widely deployed on edge devices and solve the learning tasks of incoming data streams whose class label and input feature continuously evolve, known as domain shift. Despite its prominent presence in real-world edge scenarios, existing benchmarks used by domain adaptation methods overlook evolving domains and under represent their shifts in label and feature distributions. To address this gap, we present EdgeVisionBench, a benchmark seeking to generate evolving domains of various types and reflect their realistic label and feature shifts encountered by edge-based vision applications. To facilitate evaluating domain adaptation methods on edge devices, we provide an open-source package that automates workload generation, contains popular DNN models and compression techniques, and standardizes evaluations with interactive interfaces. Code and datasets are available at https://github.com/LINC-BIT/EdgeVisionBench.

Multi-agent deep reinforcement learning (MADRL) has been widely used in many scenarios such as robotics and game AI. However, existing methods mainly focus on the optimization of agents' micro policies without considering the macro strategy. As a result, they cannot perform well in complex or sparse reward scenarios like the StarCraft Multi-Agent Challenge (SMAC) and Google Research Football (GRF). To this end, we propose a hierarchical MADRL framework called "HiMacMic" with dynamic asynchronous macro strategy. Spatially, HiMacMic determines a critical position by using a positional heat map. Temporally, the macro strategy dynamically decides its deadline and updates it asynchronously among agents. We validate HiMacMic in four widely used benchmarks, namely: Overcooked, GRF, SMAC and SMAC-v2 with nine chosen scenarios. Results show that HiMacMic not only converges faster and achieves higher results than ten existing approaches, but also shows its adaptability to different environment settings.

Active learning, a label-efficient paradigm, empowers models to interactively query an oracle for labeling new data. In the realm of LiDAR semantic segmentation, the challenges stem from the sheer volume of point clouds, rendering annotation labor-intensive and cost-prohibitive. This paper presents Annotator, a general and efficient active learning baseline, in which a voxel-centric online selection strategy is tailored to efficiently probe and annotate the salient and exemplar voxel girds within each LiDAR scan, even under distribution shift. Concretely, we first execute an in-depth analysis of several common selection strategies such as Random, Entropy, Margin, and then develop voxel confusion degree (VCD) to exploit the local topology relations and structures of point clouds. Annotator excels in diverse settings, with a particular focus on active learning (AL), active source-free domain adaptation (ASFDA), and active domain adaptation (ADA). It consistently delivers exceptional performance across LiDAR semantic segmentation benchmarks, spanning both simulation-to-real and real-to-real scenarios. Surprisingly, Annotator exhibits remarkable efficiency, requiring significantly fewer annotations, e.g., just labeling five voxels per scan in the SynLiDAR → SemanticKITTI task. This results in impressive performance, achieving 87.8% fully-supervised performance under AL, 88.5% under ASFDA, and 94.4% under ADA. We envision that Annotator will offer a simple, general, and efficient solution for label-efficient 3D applications.

The capability of generalizing to out-of-distribution data is crucial for the deployment of machine learning models in the real world. Existing domain generalization (DG) mainly embarks on offline and discrete scenarios, where multiple source domains are simultaneously accessible and the distribution shift among domains is abrupt and violent. Nevertheless, such setting may not be universally applicable to all real-world applications, as there are cases where the data distribution gradually changes over time due to various factors, e.g., the process of aging. Additionally, as the domain constantly evolves, new domains will continually emerge. Re-training and updating models with both new and previous domains using existing DG methods can be resource-intensive and inefficient. Therefore, in this paper, we present a problem formulation for Continual Domain Generalization over Temporal Drift (CDGTD). CDGTD addresses the challenge of gradually shifting data distributions over time, where domains arrive sequentially and models can only access the data of the current domain. The goal is to generalize to unseen domains that are not too far into the future. To this end, we propose an Evolving Standardization (EvoS) method, which characterizes the evolving pattern of feature distribution and mitigates the distribution shift by standardizing features with generated statistics of corresponding domain. Specifically, inspired by the powerful ability of transformers to model sequence relations, we design a multi-scale attention module (MSAM) to learn the evolving pattern under sliding time windows of different lengths. MSAM can generate statistics of current domain based on the statistics of previous domains and the learned evolving pattern. Experiments on multiple real-world datasets including images and texts validate the efficacy of our EvoS.

In recent years, the privacy-protecting framework Differential Privacy (DP) has achieved remarkable success and has been widely studied. However, there is a lack of work on DP in the area of Federated Continual Learning (FCL), which is a combination of Federated Learning (FL) and Continual Learning (CL). This paper presents a formal definition of DP-FCL and evaluates several DP-FCL methods based on Gaussian DP (GDP) and Individual DP (IDP). The experimental results indicate that gradient modification based CL strategies are not practical in DP-FCL. To the best of our knowledge, this is the first work to experimentally study DP-FCL, which can provide a reference for future research in this area.

Domain adaptive semantic segmentation attempts to make satisfactory dense predictions on an unlabeled target domain by utilizing the supervised model trained on a labeled source domain. One popular solution is self-training, which retrains the model with pseudo labels on target instances. Plenty of approaches tend to alleviate noisy pseudo labels, however, they ignore the intrinsic connection of the training data, i.e., intra-class compactness and inter-class dispersion between pixel representations across and within domains. In consequence, they struggle to handle cross-domain semantic variations and fail to build a well-structured embedding space, leading to less discrimination and poor generalization. In this work, we propose emantic-Guided Pixel Contrast (SePiCo), a novel one-stage adaptation framework that highlights the semantic concepts of individual pixels to promote learning of class-discriminative and class-balanced pixel representations across domains, eventually boosting the performance of self-training methods. Specifically, to explore proper semantic concepts, we first investigate a centroid-aware pixel contrast that employs the category centroids of the entire source domain or a single source image to guide the learning of discriminative features. Considering the possible lack of category diversity in semantic concepts, we then blaze a trail of distributional perspective to involve a sufficient quantity of instances, namely distribution-aware pixel contrast, in which we approximate the true distribution of each semantic category from the statistics of labeled source data. Moreover, such an optimization objective can derive a closed-form upper bound by implicitly involving an infinite number of (dis)similar pairs, making it computationally efficient. Extensive experiments show that SePiCo not only helps stabilize training but also yields discriminative representations, making significant progress on both synthetic-to-real and daytime-to-nighttime adaptation scenarios. The code and models are available at https://github.com/BIT-DA/SePiCo.

Partial domain adaptation (PDA) attempts to learn transferable models from a large-scale labeled source domain to a small unlabeled target domain with fewer classes, which has attracted a recent surge of interest in transfer learning. Most conventional PDA approaches endeavor to design delicate source weighting schemes by leveraging target predictions to align cross-domain distributions in the shared class space. Accordingly, two crucial issues are overlooked in these methods. First, target prediction is a double-edged sword, and inaccurate predictions will result in negative transfer inevitably. Second, not all target samples have equal transferability during the adaptation; thus, "ambiguous" target data predicted with high uncertainty should be paid more attentions. In this article, we propose a critical classes and samples discovering network (CSDN) to identify the most relevant source classes and critical target samples, such that more precise cross-domain alignment in the shared label space could be enforced by co-training two diverse classifiers. Specifically, during the training process, CSDN introduces an adaptive source class weighting scheme to select the most relevant classes dynamically. Meanwhile, based on the designed target ambiguous score, CSDN emphasizes more on ambiguous target samples with larger inconsistent predictions to enable fine-grained alignment. Taking a step further, the weighting schemes in CSDN can be easily coupled with other PDA and DA methods to further boost their performance, thereby demonstrating its flexibility. Extensive experiments verify that CSDN attains excellent results compared to state of the arts on four highly competitive benchmark datasets.

Scene recognition has attracted rising attentions of many researchers in the remote sensing fields, owing to the rapidly advancing of remote sensing devices in recent years. However, images obtained from various sensors dominate diverse sensor-specific characteristics, which will dramatically weaken the model transferability trained on a source data domain to a different target domain on account of the domain shift issues. To mitigate the domain discrepancy, most existing methods attend to align the cross-domain distributions. While the valuable knowledge of semantic relationships between different scenes is generally overlooked, and the underlying correlation across scenes cannot be fully discovered. For the sake of tackling this challenge, we propose an adaptive remote sensing scene recognition network, which can successfully transfer both the discriminative knowledge and cross-scene relationship from source to target. Specifically, in this paper, we acquire sensor-invariant representations in an adversarial manner and realize fine-grained conditional distribution alignment contrastively. In such a way, the tremendous domain gap can be mitigated to a large extent, and the discriminative and well-matched representations will be derived favorably. In addition, we explicitly construct class-wise relationship distributions belonging to two domains respectively and minimize their divergence to conduct semantic relationship knowledge transfer (SRKT), for the purpose of sufficiently unearthing the intrinsic semantic relative structures that can prompt generality of the model in the target domain. Finally, we conduct multiple experiments on representative multi-domain remote sensing benchmarks, and the extensive experimental results demonstrate the superiority of our proposed approach.

Intelligent Transportation Systems (ITS) utilises the growing trend of both communication technologies and intelligent analytics to make transportation systems more smart and efficient. Federated Learning, a privacy-preserving machine learning paradigm shows promise in being applied in this field. However, the data and device heterogeneity, and highly dynamic environment in ITS pose challenges to the performance of federated learning. One of the recent approaches to address the challenges are to choose proper participants from available clients during training. However, this research field is not fully investigated yet, and many works are still based on the classic random-based selection scheme. In this paper, we present Newt, an enhanced federated learning approach. On one hand, it includes a new client selection utility that explores the trade-off between accuracy performance in each round and system progress. On the other hand, it highlights a feedback control on the selector. Specifically, we implement a control on the selection frequency as a new dimension of client selection method design. We evaluate the proposed system with DNN training tasks on large scale FEMNIST-based datasets that are of different heterogeneity properties. The experiments show that our method outperforms the other baseline methods by as large as 20%.

Anomaly detection (AD) aims to distinguish abnormal instances from what is defined as normal, which strongly correlates with the safe and robust applications of machine learning. A well-performed anomaly detector often relies on the training on massive labeled data, while it is of high cost to annotate data in practice. Fortunately, this dilemma can be solved by transferring the knowledge of a label-rich dataset (source domain) to assist the learning on the label-scarce dataset (target domain), which is known as domain adaptation in transfer learning. In this paper, we propose a Multi-spectral Cross-domain Representation Alignment (MsRA) method for the anomaly detection in the domain adaptation setting, where we can only access normal source data andlimitednormal target data. Specifically, MsRA first constructs multi-spectral feature representations by fusing different frequency components of the original features, which mitigates the information scarcity due to limited target training data by capturing richer input pattern information. Then we employ the adversarial training strategy to learn domain-invariant features and force the features of normal data to be more compact by the center clustering. Finally, the distance of each sample to the prototype of normal class can be used as its anomaly score, where the prototype is the center of both source and target data. In this way, we achieve anomaly detection in an end-to-end manner, without two-stage training for feature extraction and anomaly detection. Comprehensive experiments on cross-domain anomaly detection benchmarks validate the effectiveness of MsRA.

Recommender systems are playing a significant role in modern society to alleviate the information/choice overload problem, since Internet users may feel hard to identify the most favorite items or products from millions of candidates. Thanks to the recent successes in computer vision, auxiliary learning has become a powerful means to improve the performance of a target (primary) task. Even though helpful, the auxiliary learning scheme is still less explored in recommendation models. To integrate the auxiliary learning scheme, we propose a novel meta auxiliary learning framework to facilitate the recommendation model training, i.e., user and item latent representations. Specifically, we construct two self-supervised learning tasks, regarding both users and items, as auxiliary tasks to enhance the representation effectiveness of users and items. Then the auxiliary and primary tasks are further modeled as a meta learning paradigm to adaptively control the contribution of auxiliary tasks for improving the primary recommendation task. This is achieved by an implicit gradient method guaranteeing less time complexity compared with conventional meta learning methods. Via a comparison using four real-world datasets with a number of state-of-the-art methods, we show that the proposed model outperforms the best existing models on the Top-K recommendation by 3% to 23%.

Unsupervised domain adaptation enables knowledge transfer from a labeled source domain to an unlabeled target domain by reducing the cross-domain distribution discrepancy, and the adversarial learning based paradigm has achieved remarkable success. On top of this, recent works seeks to further regularize the classification decision boundary via self-training to learn target adaptive classifier with pseudo-labeled target samples. However, since pseudo labels are inevitably noisy, most of prior methods focus on manually designing elaborate target selection algorithms or optimization objectives. Different from them, we propose a meta-learning based target-reweighting regularization algorithm called MetaReg. Specifically, MetaReg is motivated by the intuition that an ideal target classifier trained on correct target pseudo labels should make small classification errors on target-like source samples. Therefore, we explicitly define a meta reweighting problem that aims to find optimal weights for different samples by minimizing the classification loss on a class-balanced set consisting of source samples that are most similar to target ones. The optimization problem is solved efficiently with a simplified approximation technique. As a result, the automatically learned optimal weights are utilized to reweight pseudo-labeled target samples and regularize the model learning. Comprehensive experiments verify that MetaReg outperforms the non-regularized UDA counterparts with state-of-the-art performance.

Unsupervised domain adaptation (UDA) aims to utilize knowledge from a label-rich source domain to understand a similar yet distinct unlabeled target domain. Notably, global distribution statistics across domains and local semantic characteristics across samples, are two essential factors of data analysis that should be fully explored. Most existing UDA approaches either harness only one of them or fail to closely associate them for efficient adaptation. In this work, we propose a unified framework, called Collaborative Alignment Framework (CAF), which simultaneously reduces the global domain discrepancy and preserves the local semantic consistency for cross-domain knowledge transfer in a collaborative manner. Specifically, for domain-oriented alignment, we utilize adversarial training or minimize the Wasserstein distance between the two distributions to learn domain-level invariant representations. For semantic-oriented matching, we capture the semantic discrepancy between the predictions of two diverse task-specific classifiers and enhance the features of target data to be near the support of the source data class-wisely, which promotes semantic consistency across domains effectively. These two adaptation processes can be deeply intertwined in CAF via collaborative training, thus CAF can learn domain-invariant and semantic-consistent feature representations. Extensive experiments on four popular benchmarks, including DomainNet, VisDA-2017, Office-31, and ImageCLEF, demonstrate the proposed methods significantly outperform the existing methods, especially on the large-scale dataset. The code is available athttps://github.com/BIT-DA/CAF.

Mobile crowdsensing (MCS) by unmanned aerial vehicles (UAVs) servicing delay-sensitive applications becomes popular by navigating a group of UAVs to take advantage of their equipped high-precision sensors and durability for data collection in harsh environments. In this paper, we aim to simultaneously maximize collected data amount, geographical fairness, and minimize the energy consumption of all UAVs, as well as to guarantee the data freshness by setting a deadline in each timeslot. Specifically, we propose a centralized control, distributed execution framework by decentralized deep reinforcement learning (DRL) for delay-sensitive and energy-efficient UAV crowdsensing, called “DRL-eFresh”. It includes a synchronous computational architecture with GRU sequential modeling to generate multi-UAV navigation decisions. Also, we derive an optimal time allocation solution for data collection while considering all UAV efforts and avoiding much data dropout due to limited data upload time and wireless data rate. Simulation results show that DRL-eFresh significantly improves the energy efficiency, as compared to the best baseline DPPO, by 14% and 22% on average when varying different sensing ranges and number of PoIs, respectively.

Tabular visualization techniques integrate visual representations with tabular data to avoid additional cognitive load caused by splitting users' attention. However, most of the existing studies focus on simple flat tables instead of hierarchical tables, whose complex structure limits the expressiveness of visualization results and affects users' efficiency in visualization construction. We present HiTailor, a technique for presenting and exploring hierarchical tables. HiTailor constructs an abstract model, which defines row/column headings as biclustering and hierarchical structures. Based on our abstract model, we identify three pairs of operators, Swap/Transpose, ToStacked/ToLinear, Fold/Unfold, for transformations of hierarchical tables to support users' comprehensive explorations. After transformation, users can specify a cell or block of interest in hierarchical tables as a TableUnit for visualization, and HiTailor recommends other related TableUnits according to the abstract model using different mechanisms. We demonstrate the usability of the HiTailor system through a comparative study and a case study with domain experts, showing that HiTailor can present and explore hierarchical tables from different viewpoints. HiTailor is available at https://github.com/bitvis2021/HiTailor.

Unsupervised domain adaptation has recently emerged as an effective paradigm for generalizing deep neural networks to new target domains. However, there is still enormous potential to be tapped to reach the fully supervised performance. In this paper, we present a novel active learning strategy to assist knowledge transfer in the target domain, dubbed active domain adaptation. We start from an observation that energy-based models exhibit free energy biases when training (source) and test (target) data come from different distributions. Inspired by this inherent mechanism, we empirically reveal that a simple yet efficient energy-based sampling strategy sheds light on selecting the most valuable target samples than existing approaches requiring particular architectures or computation of the distances. Our algorithm, Energy-based Active Domain Adaptation (EADA), queries groups of target data that incorporate both domain characteristic and instance uncertainty into every selection round. Meanwhile, by aligning the free energy of target data compact around the source domain via a regularization term, domain gap can be implicitly diminished. Through extensive experiments, we show that EADA surpasses state-of-the-art methods on well-known challenging benchmarks with substantial improvements, making it a useful option in the open world. Code is available at https://github.com/BIT-DA/EADA.

Vision-based autonomous urban driving in dense traffic is quite challenging due to the complicated urban environment and the dynamics of the driving behaviors. Widely-applied methods either heavily rely on hand-crafted rules or learn from limited human experience, which makes them hard to generalize to rare but critical scenarios. In this paper, we present a novel CAscade Deep REinforcement learning framework, CADRE, to achieve model-free vision-based autonomous urban driving. In CADRE, to derive representative latent features from raw observations, we first offline train a Co-attention Perception Module (CoPM) that leverages the co-attention mechanism to learn the inter-relationships between the visual and control information from a pre-collected driving dataset. Cascaded by the frozen CoPM, we then present an efficient distributed proximal policy optimization framework to online learn the driving policy under the guidance of particularly designed reward functions. We perform a comprehensive empirical study with the CARLA NoCrash benchmark as well as specific obstacle avoidance scenarios in autonomous urban driving tasks. The experimental results well justify the effectiveness of CADRE and its superiority over the state-of-the-art by a wide margin.

Domain generalization (DG) is essentially an out-of-distribution problem, aiming to generalize the knowledge learned from multiple source domains to an unseen target domain. The mainstream is to leverage statistical models to model the dependence between data and labels, intending to learn representations independent of domain. Nevertheless, the statistical models are superficial descriptions of reality since they are only required to model dependence instead of the intrinsic causal mechanism. When the dependence changes with the target distribution, the statistic models may fail to generalize. In this regard, we introduce a general structural causal model to formalize the DG problem. Specifically, we assume that each input is constructed from a mix of causal factors (whose relationship with the label is invariant across domains) and non-causal factors (category-independent), and only the former cause the classification judgments. Our goal is to extract the causal factors from inputs and then reconstruct the invariant causal mechanisms. However, the theoretical idea is far from practical of DG since the required causal/non-causal factors are unobserved. We highlight that ideal causal factors should meet three basic properties: separated from the non-causal ones, jointly independent, and causally sufficient for the classification. Based on that, we propose a Causality Inspired Representation Learning (CIRL) algorithm that enforces the representations to satisfy the above properties and then uses them to simulate the causal factors, which yields improved generalization ability. Extensive experimental results on several widely used datasets verify the effectiveness of our approach.11Code is available at “https://github.com/BIT-DA/CIRL”.

Self-training has greatly facilitated domain adaptive semantic segmentation, which iteratively generates pseudo labels on unlabeled target data and retrains the network. However, realistic segmentation datasets are highly imbalanced, pseudo labels are typically biased to the majority classes and basically noisy, leading to an error-prone and suboptimal model. In this paper, we propose a simple region-based active learning approach for semantic segmentation under a domain shift, aiming to automatically query a small partition of image regions to be labeled while maximizing segmentation performance. Our algorithm, Region Impurity and Prediction Uncertainty (RIPU), introduces a new acquisition strategy characterizing the spatial adjacency of image regions along with the prediction confidence. We show that the proposed region-based selection strategy makes more efficient use of a limited budget than image-based or point-based counterparts. Further, we enforce local prediction consistency between a pixel and its nearest neighbors on a source image. Alongside, we develop a negative learning loss to make the features more discriminative. Extensive experiments demonstrate that our method only requires very few annotations to almost reach the supervised performance and substantially outperforms state-of-the-art methods. The code is available at https://github.com/BIT-DA/RIPU.

Spatial crowdsourcing (SC) has been proved quite successful by employing human participants to achieve certain tasks like Uber and Gigwalk. Meanwhile, with the fast devel-opment of unmanned aerial vehicles (e.g., drones), they have become a new source of data collectors equipped with a variety of different sensors. In this paper, we propose a novel SC scenario, enabling human participants to work collaboratively with drones in the presence of multiple charging stations to achieve certain data collection tasks, like videography and surveillance. We propose a novel deep reinforcement learning (D RL) framework called “FD- MAPPO (Cubic Map)”, which consists of a fully de-centralized multi-agent DRL (MADRL) algorithm called “Fully Decentralized Multi-Agent Proximal Policy Optimization (FD-MAPPO)”, and a spatiotemporal memory augmented neural network with novel cubic writing and spatially contextual reading mechanisms called “Cubic Map”. Cubic Map extracts long-term spatiotemporal features, navigates drones to accurately locate the position of the target, i.e., charging stations or sensors. Extensive results on two real datasets of KAIST and NCSU campuses show that FD- MAPPO (Cubic Map) consistently outperforms six other baselines in terms of efficiency.

With the current large scale deployment of machine learning technologies, such as those on cloud servers and edge and IoT hardwares, machine learning systems have been widely prevalence. Practical requirement has driven their performance increase in both academia and industry. However, the application requirement varies greatly across different applications, and directly using off-the-shelf systems might not be sufficient in many cases. In this work, we first propose to implement a series of techniques to optimize performance of convolution operation, one of the most important operations, in constructing deep learning networks. Besides, we also propose to apply the automated empirical optimisation of software approach to improve the performance of operators in machine learning system, most notably across various hardware platforms. Evaluation compared to existing libraries on different hardware devices has proved the efficiency of our proposed method.

Mobile Crowdsensing (MCS) with smart devices has become an appealing paradigm for urban sensing. With the development of 5G-and-beyond technologies, unmanned aerial vehicles (UAVs) become possible for real-time applications, including wireless coverage, search and even disaster response. In this paper, we consider to use a group of UAVs as aerial base stations (BSs) to move around and collect data from multiple MCS users, forming a UAV crowdsensing campaign (UCS). Our goal is to maximize the collected data, geographical coverage whiling minimizing the age-of-information (AoI) of all mobile users simultaneously, with efficient use of constrained energy reserve. We propose a model-based deep reinforcement learning (DRL) framework called "GCRL-min(AoI)", which mainly consists of a novel model-based Monte Carlo tree search (MCTS) structure based on state-of-the-art approach MCTS (AlphaZero). We further improve it by adding a spatial UAV-user correlation extraction mechanism by a relational graph convolutional network (RGCN), and a next state prediction module to reduce the dependance of experience data. Extensive results and trajectory visualization on three real human mobility datasets in Purdue University, KAIST and NCSU show that GCRL-min(AoI) consistently outperforms five baselines, when varying different number of UAVs and maximum coupling loss in terms of four metrics.

Edge-cloud jobs are rapidly prevailing in many application domains, posing the challenge of using both resource-strenuous edge devices and elastic cloud resources. Efficient resource allocation on such jobs via scheduling algorithms is essential to guarantee their performance, e.g. latency. Deep reinforcement learning (DRL) is increasingly adopted to make scheduling decisions but faces the conundrum of achieving high rewards at a low training overhead. It is unknown if such a DRL can be applied to timely tune the scheduling algorithms that are adopted in response to fast changing workloads and resources. In this paper, we propose EdgeTuner to effectively leverage DRL to select scheduling algorithms online for edge-cloud jobs. The enabling features of EdgeTuner are sophisticated DRL model that captures complex dynamics of Edge-Cloud jobs/tasks and an effective simulator to emulate the response times of short-running jobs in accordance to dynamically changing scheduling algorithms. EdgeTuner trains DRL agents offline by directly interacting with the simulator. We implement EdgeTuner on Kubernetes scheduler and extensively evaluate it on Kubernetes cluster testbed driven by the production traces. Our results show that EdgeTuner outperforms prevailing scheduling algorithms by achieving significant lower job response time while accelerating DRL training speed by more than 180x.

DETR-style detectors stand out amongst in-domain scenarios, but their properties in domain shift settings are under-explored. This paper aims to build a simple but effective baseline with a DETR-style detector on domain shift settings based on two findings. For one, mitigating the domain shift on the backbone and the decoder output features excels in getting favorable results. For another, advanced domain alignment methods in both parts further enhance the performance. Thus, we propose the Object-Aware Alignment (OAA) module and the Optimal Transport based Alignment (OTA) module to achieve comprehensive domain alignment on the outputs of the backbone and the detector. The OAA module aligns the foreground regions identified by pseudo-labels in the backbone outputs, leading to domain-invariant base features. The OTA module utilizes sliced Wasserstein distance to maximize the retention of location information while minimizing the domain gap in the decoder outputs. We implement the findings and the alignment modules into our adaptation method, and it benchmarks the DETR-style detector on the domain shift settings. Experiments on various domain adaptive scenarios validate the effectiveness of our method.

Extensive studies on Unsupervised Domain Adaptation (UDA) have propelled the deployment of deep learning from limited experimental datasets into real-world unconstrained domains. Most UDA approaches align features within a common embedding space and apply a shared classifier for target prediction. However, since a perfectly aligned feature space may not exist when the domain discrepancy is large, these methods suffer from two limitations. First, the coercive domain alignment deteriorates target domain discriminability due to lacking target label supervision. Second, the source-supervised classifier is inevitably biased to source data, thus it may underperform in target domain. To alleviate these issues, we propose to simultaneously conduct feature alignment in two individual spaces focusing on different domains, and create for each space a domain-oriented classifier tailored specifically for that domain. Specifically, we design a Domain-Oriented Transformer (DOT) that has two individual classification tokens to learn different domain-oriented representations, and two classifiers to preserve domain-wise discriminability. Theoretical guaranteed contrastive-based alignment and the source-guided pseudo-label refinement strategy are utilized to explore both domain-invariant and specific information. Comprehensive experiments validate that our method achieves state-of-the-art on several benchmarks. Code is released at https://github.com/BIT-DA/Domain-Oriented-Transformer.

Infrared cameras are often utilized to enhance the night vision since the visible light cameras exhibit inferior efficacy without sufficient illumination. However, infrared data possesses inadequate color contrast and representation ability attributed to its intrinsic heat-related imaging principle, which hinders its application. Although, the domain gaps between unpaired nighttime infrared and daytime visible videos are even huger than paired ones that captured at the same time, establishing an effective translation mapping will greatly contribute to various fields. In this case, the structural knowledge within nighttime infrared videos and semantic information contained in the translated daytime visible pairs could be utilized simultaneously. To this end, we propose a tailored framework ROMA that couples with our introduced cRoss-domain regiOn siMilarity mAtching technique for bridging the huge gaps. To be specific, ROMA could efficiently translate the unpaired nighttime infrared videos into fine-grained daytime visible ones, meanwhile maintain the spatiotemporal consistency via matching the cross-domain region similarity. Furthermore, we design a multiscale region-wise discriminator to distinguish the details from synthesized visible results and real references. Moreover, we provide a new and challenging dataset encouraging further research for unpaired nighttime infrared and daytime visible video translation, named InfraredCity, which is $20$ times larger than the recently released infrared-related dataset IRVI. Codes and datasets are available https://github.com/BIT-DA/ROMA here.

Domain adaptation (DA) attempts to transfer knowledge learned in the labeled source domain to the unlabeled but related target domain without requiring large amounts of target supervision. Recent advances in DA mainly proceed by aligning the source and target distributions. Despite the significant success, the adaptation performance still degrades accordingly when the source and target domains encounter a large distribution discrepancy. We consider this limitation may attribute to the insufficient exploration of domain-specialized features because most studies merely concentrate on domain-general feature learning in task-specific layers and integrate totally-shared convolutional networks (convnets) to generate common features for both domains. In this paper, we relax the completely-shared convnets assumption adopted by previous DA methods and propose Domain Conditioned Adaptation Network (DCAN), which introduces domain conditioned channel attention module with a multi-path structure to separately excite channel activation for each domain. Such a partially-shared convnets module allows domain-specialized features in low-level to be explored appropriately. Further, given the knowledge transferability varying along with convolutional layers, we develop Generalized Domain Conditioned Adaptation Network (GDCAN) to automatically determine whether domain channel activations should be separately modeled in each attention module. Afterward, the critical domain-specialized knowledge could be adaptively extracted according to the domain statistic gaps. As far as we know, this is the first work to explore the domain-wise convolutional channel activations separately for deep DA networks. Additionally, to effectively match high-level feature distributions across domains, we consider deploying feature adaptation blocks after task-specific layers, which can explicitly mitigate the domain discrepancy. Extensive experiments on four cross-domain benchmarks, including DomainNet, Office-Home, Office-31, and ImageCLEF, demonstrate the proposed approaches outperform the existing methods by a large margin, especially on the large-scale challenging dataset. The code and models are available at https://github.com/BIT-DA/GDCAN.

Few-shot segmentation aims at learning to segment query images guided by only a few annotated images from the support set. Previous methods rely on mining the feature embedding similarity across the query and the support images to achieve successful segmentation. However, these models tend to perform badly in cases where the query instances have a large variance from the support ones. To enhance model robustness against such intra-class variance, we propose a Double Recalibration Network (DRNet) with two recalibration modules, i.e., the Self-adapted Recalibration (SR) module and the Cross-attended Recalibration (CR) module. In particular, beyond learning robust feature embedding for pixel-wise comparison between support and query as in conventional methods, the DRNet further exploits semantic-aware knowledge embedded in the query image to help segment itself, which we call ‘self-adapted recalibration’. More specifically, DRNet first employs guidance from the support set to roughly predict an incomplete but correct initial object region for the query image, and then reversely uses the feature embedding extracted from the incomplete object region to segment the query image. Also, we devise a CR module to refine the feature representation of the query image by propagating the underlying knowledge embedded in the support image’s foreground to the query. Instead of foreground global pooling, we refine the response at each pixel in the query feature map by attending to all foreground pixels in the support feature map and taking the weighted average by their similarity; meanwhile, feature maps of the query image are also added back to weighted feature maps as a residual connection. Our DRNet can effectively address the intra-class variance under the few-shot setting with such two recalibration modules, and mine more accurate target regions for query images. We conduct extensive experiments on the popular benchmarks PASCAL-$5^{i}$and COCO-$20^{i}$. The DRNet with the best configuration achieves the mIoU of$\textbf {63.6}\%$and$\textbf {64.9}\%$on PASCAL-$5^{i}$and$\textbf {44.7}\%$and$\textbf {49.6}\%$on COCO-$20^{i}$for 1-shot and 5-shot settings respectively, significantly outperforming the state-of-the-arts without any bells and whistles. Code is available at:https://github.com/fangzy97/drnet.

Personalized location prediction is key to many mobile applications and services. In this paper, motivated by both statistical and visualized preliminary analysis on three real datasets, we observe a strong spatiotemporal correlation for user trajectories among the visited area-of-interests (AoIs) and different time periods on both weekly and daily basis, which directly motivates our time-aware location prediction model design called “$t$t-LocPred”. It models the spatial correlations among AoIs by coarse-grained convolutional processing of the user trajectories in AoIs of different time periods (“ConvAoI”); and predicts his/her fine-grained next visited PoI using a novel memory-augmented attentive LSTM model (“mem-attLSTM”) to capture long-term behavior patterns. Experimental results show that$t$t-LocPred outperforms 8 baselines. We also show the impact of hyperparameters and the benefits ConvAoI can bring to these baselines.

Deep neural networks (DNNs) have been showing significant success in various anomaly detection applications such as smart surveillance and industrial quality control. It is increasingly important to detect anomalies directly on edge devices, because of high responsiveness requirements and tight latency constraints. The accuracy of DNN-based solutions rely on large model capacity and thus long training and inference time, making them inapplicable on resource strenuous edge devices. It is hence imperative to scale DNN model sizes in correspondence to the run-time system requirements, i.e. meeting deadlines with minimal accuracy losses, which are highly dependent on the platforms and real-time system status. Existing scaling techniques either take long training time to pre-generate scaling options or disturb the unsteady training process of anomaly detection DNNs, lacking the adaptability to heterogeneous edge systems and incurring low inference accuracies. In this paper, we present LightDNN to scale DNN models for anomaly detection applications at edge, featuring high detection accuracies with lightweight training and inference time. To this end, LightDNN quickly extracts and compresses blocks in a DNN, and provides large scaling space (e.g. 1 million options) by dynamically combining these compressed blocks online. At run-time, LightDNN predicts the DNN's inference latency according to the monitored system status, and optimizes the combination of blocks to maximize its accuracy under deadline constraints. We implement and extensively evaluate LightDNN on both CPU and GPU edge platforms using 8 popular anomaly detection workloads. Comparative experiments with state-of-the-art methods show that our approach provides 145.8 to 0.56 trillion times more scaling options without increasing training and inference overheads, thus achieving as much as 15.05% increase in accuracy under the same deadlines.

With the massive amount of data generated from mobile devices and the increase of computing power of edge devices, the paradigm of Federated Learning has attracted great momentum. In federated learning, distributed and heterogeneous nodes collaborate to learn model parameters. However, while providing benefits such as privacy by design and reduced latency, the heterogeneous network present challenges to the synchronisation methods, or barrier control methods, used in training, regarding system progress and model convergence etc. The design of these barrier mechanisms is critical for the performance and scalability of federated learning systems. We propose a new barrier control technique called Probabilistic Synchronous Parallel (PSP). In contrast to existing mechanisms, it introduces a sampling primitive that composes with existing barrier control mechanisms to produce a family of mechanisms with improved convergence speed and scalability. Our proposal is supported with a convergence analysis of PSP-based SGD algorithm. In practice, we also propose heuristic techniques that further improve the efficiency of PSP. We evaluate the performance of proposed methods using the federated learning specific FEMNSIT dataset. The evaluation results show that PSP can effectively achieve good balance between system efficiency and model accuracy, mitigating the challenge of heterogeneity in federated learning.

Unsupervised domain adaptation challenges the problem of transferring knowledge from a well-labelled source domain to an unlabelled target domain. Recently, adversarial learning with bi-classifier has been proven effective in pushing cross-domain distributions close. Prior approaches typically leverage the disagreement between bi-classifier to learn transferable representations, however, they often neglect the classifier determinacy in the target domain, which could result in a lack of feature discriminability. In this paper, we present a simple yet effective method, namely Bi-Classifier Determinacy Maximization (BCDM), to tackle this problem. Motivated by the observation that target samples cannot always be separated distinctly by the decision boundary, here in the proposed BCDM, we design a novel classifier determinacy disparity (CDD) metric, which formulates classifier discrepancy as the class relevance of distinct target predictions and implicitly introduces constraint on the target feature discriminability. To this end, the BCDM can generate discriminative representations by encouraging target predictive outputs to be consistent and determined, meanwhile, preserve the diversity of predictions in an adversarial manner. Furthermore, the properties of CDD as well as the theoretical guarantees of BCDM's generalization bound are both elaborated. Extensive experiments show that BCDM compares favorably against the existing state-of-the-art domain adaptation methods.

Real-world training data usually exhibits long-tailed distribution, where several majority classes have a significantly larger number of samples than the remaining minority classes. This imbalance degrades the performance of typical supervised learning algorithms designed for balanced training sets. In this paper, we address this issue by augmenting minority classes with a recently proposed implicit semantic data augmentation (ISDA) algorithm [37], which produces diversified augmented samples by translating deep features along many semantically meaningful directions. Importantly, given that ISDA estimates the class-conditional statistics to obtain semantic directions, we find it ineffective to do this on minority classes due to the insufficient training data. To this end, we propose a novel approach to learn transformed semantic directions with meta-learning automatically. In specific, the augmentation strategy during training is dynamically optimized, aiming to minimize the loss on a small balanced validation set, which is approximated via a meta update step. Extensive empirical results on CIFAR-LT-10/100, ImageNet-LT, and iNaturalist 2017/2018 validate the effectiveness of our method.

Domain adaptation has been widely explored by transferring the knowledge from a label-rich source domain to a related but unlabeled target domain. Most existing domain adaptation algorithms attend to adapting feature representations across two domains with the guidance of a shared source-supervised classifier. However, such classifier limits the generalization ability towards unlabeled target recognition. To remedy this, we propose a Transferable Semantic Augmentation (TSA) approach to enhance the classifier adaptation ability through implicitly generating source features towards target semantics. Specifically, TSA is inspired by the fact that deep feature transformation towards a certain direction can be represented as meaningful semantic altering in the original input space. Thus, source features can be augmented to effectively equip with target semantics to train a more transferable classifier. To achieve this, for each class, we first use the inter-domain feature mean difference and target intra-class feature covariance to construct a multivariate normal distribution. Then we augment source features with random directions sampled from the distribution class-wisely. Interestingly, such source augmentation is implicitly implemented through an expected transferable cross-entropy loss over the augmented source distribution, where an upper bound of the expected loss is derived and minimized, introducing negligible computational overhead. As a light-weight and general technique, TSA can be easily plugged into various domain adaptation methods, bringing remarkable improvements. Comprehensive experiments on cross-domain benchmarks validate the efficacy of TSA.

Domain adaptation (DA) enables knowledge transfer from a labeled source domain to an unlabeled target domain by reducing the cross-domain distribution discrepancy. Most prior DA approaches leverage complicated and powerful deep neural networks to improve the adaptation capacity and have shown remarkable success. However, they may have a lack of applicability to real-world situations such as real-time interaction, where low target inference latency is an essential requirement under limited computational budget. In this paper, we tackle the problem by proposing a dynamic domain adaptation (DDA) framework, which can simultaneously achieve efficient target inference in low-resource scenarios and inherit the favorable cross-domain generalization brought by DA. In contrast to static models, as a simple yet generic method, DDA can integrate various domain confusion constraints into any typical adaptive network, where multiple intermediate classifiers can be equipped to infer “easier” and “harder” target data dynamically. Moreover, we present two novel strategies to further boost the adaptation performance of multiple prediction exits: 1) a confidence score learning strategy to derive accurate target pseudo labels by fully exploring the prediction consistency of different classifiers; 2) a class-balanced self-training strategy to explicitly adapt multi-stage classifiers from source to target without losing prediction diversity. Extensive experiments on multiple benchmarks are conducted to verify that DDA can consistently improve the adaptation performance and accelerate target inference under domain shift and limited resources scenarios.

Weakly Supervised Semantic Segmentation (WSSS) with image-level annotation uses class activation maps from the classifier as pseudo-labels for semantic segmentation. However, such activation maps usually highlight the local discriminative regions rather than the whole object, which deviates from the requirement of semantic segmentation. To explore more comprehensive class-specific activation maps, we propose an Embedded Discriminative Attention Mechanism (EDAM) by integrating the activation map generation into the classification network directly for WSSS. Specifically, a Discriminative Activation (DA) layer is designed to explicitly produce a series of normalized class-specific masks, which are then used to generate class-specific pixel-level pseudo-labels demanded in segmentation. For learning the pseudo-labels, the masks are multiplied with the feature maps after the backbone to generate the discriminative activation maps, each of which encodes the specific information of the corresponding category in the input images. Given such class-specific activation maps, a Collaborative Multi-Attention (CMA) module is proposed to extract the collaborative information of each given category from images in a batch. In inference, we directly use the activation masks from the DA layer as pseudo-labels for segmentation. Based on the generated pseudo-labels, we achieve the mIoU of 70.60% on PASCAL VOC 2012 segmentation testset, which is the new state-of-the-art, to our best knowledge. Code and pre-trained models are available online soon.

Domain adaptation (DA) paves the way for label annotation and dataset bias issues by the knowledge transfer from a label-rich source domain to a related but unlabeled target domain. A mainstream of DA methods is to align the feature distributions of the two domains. However, the majority of them focus on the entire image features where irrelevant semantic information, e.g., the messy background, is inevitably embedded. Enforcing feature alignments in such case will negatively influence the correct matching of objects and consequently lead to the semantically negative transfer due to the confusion of irrelevant semantics. To tackle this issue, we propose Semantic Concentration for Domain Adaptation (SCDA), which encourages the model to concentrate on the most principal features via the pair-wise adversarial alignment of prediction distributions. Specifically, we train the classifier to class-wisely maximize the prediction distribution divergence of each sample pair, which enables the model to find the region with large differences among the same class of samples. Meanwhile, the feature extractor attempts to minimize that discrepancy, which suppresses the features of dissimilar regions among the same class of samples and accentuates the features of principal parts. As a general method, SCDA can be easily integrated into various DA methods as a regularizer to further boost their performance. Extensive experiments on the cross-domain benchmarks show the efficacy of SCDA.

Understanding the movement patterns of humans and vehicles traveling in a city is important for many applications like emergency evacuation and rescue, as well as city planning and management. In this paper, we aim to predict citywide crowd flows within a period in the future to give aid to urban management, through modeling spatiotemporal patterns of recent crowd flows. We present a novel deep model for this task, called "AttConvLSTM", which leverages a convolutional LSTM (ConvLSTM), Convolutional Neural Networks (CNNs) along with an attention mechanism, where ConvLSTM keeps spatial information as intact as possible during sequential analysis, and the attention mechanism can focus important crowd flow variations which cannot be identified by the recurrent module. We conducted extensive experiments for performance evaluation using three large datasets, including Beijing Taxi dataset, Rome Taxi dataset, and Chengdu Didi chauffeuring trace. The experimental results show that AttConvLSTM significantly outperforms several widely-used baselines in terms of Root Mean Squared Error (RMSE), and Mean Average Percentage Error (MAPE), indicating that our approach can deal with crowd flows with different dynamics in both spatial and temporal domains, and make valid predictions several steps ahead.

Ticket pricing is always a challenging problem for world-wide airline companies when balancing their revenues and sales, where tickets are often discounted to adapt to a marketable price level. In this paper, we transform the problem of modeling Passenger Payment Willingness (PPW) into a top-$K$recommendation problem, where a list of ticket discounted ratios is recommended by fully considering ticket discount histories of peer airline companies and multi-dimensional ticket attributes, i.e., passenger purchasing capability. We propose a novel deep model, called “NCL”, which integrates N-Beats, a Graph Convolutional Neural Network (GCN) and an LSTM together to model temporal variations of ticket discounts and complex relationships among multi-dimensional ticket attributes. Specifically, first, the ticket discount historical sequence is integrated by N-Beats. Then, multi-dimensional ticket attributes are divided into dynamic and static categories, where an attribute graph of static attributes is constructed, and a GCN is leveraged to extract features from it. After, LSTM is used to perform temporal feature fusion on the dynamic attributes. Finally, NCL integrates features from all the above and predicts future ticket discounts. Experiments confirm that the prediction accuracy of NCL is more than 60% in terms of ACC@1.

Data collection by mobile crowdsensing (MCS) is emerging as data sources for smart city applications, however how to ensure data freshness has sparse research exposure but quite important in practice. In this paper, we consider to use a group of mobile agents (MAs) like UAVs and driverless cars which are equipped with multiple antennas to move around in the task area to collect data from deployed sensor nodes (SNs). Our goal is to minimize the age of information (AoI) of all SNs and energy consumption of MAs during movement and data upload. To this end, we propose a centralized deep reinforcement learning (DRL)-based solution called "DRL-freshMCS" for controlling MA trajectory planning and SN scheduling. We further utilize implicit quantile networks to maintain the accurate value estimation and steady policies for MAs. Then, we design an exploration and exploitation mechanism by dynamic distributed prioritized experience replay. We also derive the theoretical lower bound for episodic AoI. Extensive simulation results show that DRL-freshMCS significantly reduces the episodic AoI per remaining energy, compared to five baselines when varying different number of antennas and data upload thresholds, and number of SNs. We also visualize their trajectories and AoI update process for clear illustrations.

Fast and efficient access to environmental and life data is key to the successful disaster response. Vehicular crowdsourcing (VC) by a group of unmanned vehicles (UVs) like drones and unmanned ground vehicles to collect these data from Point-of-Interests (PoIs) e.g., possible survivor spots and fire site, provides an efficient way to assist disaster rescue. In this paper, we explicitly consider to navigate a group of UVs in a 3-dimensional (3D) disaster workzone to maximize the amount of collected data, geographical fairness, energy efficiency, while minimizing data dropout due to limited transmission rate. We propose DRL-DisasterVC(3D), a distributed deep reinforcement learning framework, with a repetitive experience replay (RER) to improve learning efficiency, and a clipped target network to increase learning stability. We also use a 3D convolutional neural network (3D CNN) with multi-head-relational attention (MHRA) for spatial modeling, and add auxiliary pixel control (PC) for spatial exploration. We designed a novel disaster response simulator, called "DisasterSim", and conduct extensive experiments to show that DRL-DisasterVC(3D) outperforms all five baselines in terms of energy efficiency when varying the numbers of UVs, PoIs and SNR threshold.

Human vision is often adversely affected by complex environmental factors, especially in night vision scenarios. Thus, infrared cameras are often leveraged to help enhance the visual effects via detecting infrared radiation in the surrounding environment, but the infrared videos are undesirable due to the lack of detailed semantic information. In such a case, an effective video-to-video translation method from the infrared domain to the visible light counterpart is strongly needed by overcoming the intrinsic huge gap between infrared and visible fields. To address this challenging problem, we propose an infrared-to-visible (I2V) video translation method I2V-GAN to generate fine-grained and spatial-temporal consistent visible light videos by given unpaired infrared videos. Technically, our model capitalizes on three types of constraints: 1) adversarial constraint to generate synthetic frames that are similar to the real ones, 2) cyclic consistency with the introduced perceptual loss for effective content conversion as well as style preservation, and 3) similarity constraints across and within domains to enhance the content and motion consistency in both spatial and temporal spaces at a fine-grained level. Furthermore, the current public available infrared and visible light datasets are mainly used for object detection or tracking, and some are composed of discontinuous images which are not suitable for video tasks. Thus, we provide a new dataset for infrared-to-visible video translation, which is named IRVI. Specifically, it has 12 consecutive video clips of vehicle and monitoring scenes, and both infrared and visible light videos could be apart into 24352 frames. Comprehensive experiments on IRVI validate that I2V-GAN is superior to the compared state-of-the-art methods in the translation of infrared-to-visible videos with higher fluency and finer semantic details. Moreover, additional experimental results on the flower-to-flower dataset indicate I2V-GAN is also applicable to other video translation tasks. The code and IRVI dataset are available at https://github.com/BIT-DA/I2V-GAN.

Deep neural networks (DNNs) have become ubiquitous techniques in mobile and embedded systems for applications such as image/object recognition and classification. The trend of executing multiple DNNs simultaneously exacerbate the existing limitations of meeting stringent latency/accuracy requirements on resource constrained mobile devices. The prior art sheds light on exploring the accuracy-resource tradeoff by scaling the model sizes in accordance to resource dynamics. However, such model scaling approaches face to imminent challenges: (i) large space exploration of model sizes, and (ii) prohibitively long training time for different model combinations. In this paper, we present LegoDNN, a lightweight, block-grained scaling solution for running multi-DNN workloads in mobile vision systems. LegoDNN guarantees short model training times by only extracting and training a small number of common blocks (e.g. 5 in VGG and 8 in ResNet) in a DNN. At run-time, LegoDNN optimally combines the descendant models of these blocks to maximize accuracy under specific resources and latency constraints, while reducing switching overhead via smart block-level scaling of the DNN. We implement LegoDNN in TensorFlow Lite and extensively evaluate it against state-of-the-art techniques (FLOP scaling, knowledge distillation and model compression) using a set of 12 popular DNN models. Evaluation results show that LegoDNN provides 1,296x to 279,936x more options in model sizes without increasing training time, thus achieving as much as 31.74% improvement in inference accuracy and 71.07% reduction in scaling energy consumptions.

Domain adaptation (DA) attempts to transfer the knowledge from a labeled source domain to an unlabeled target domain that follows different distribution from the source. To achieve this, DA methods include a source classification objective to extract the source knowledge and a domain alignment objective to diminish the domain shift, ensuring knowledge transfer. Typically, former DA methods adopt some weight hyper-parameters to linearly combine the training objectives to form an overall objective. However, the gradient directions of these objectives may conflict with each other due to domain shift. Under such circumstances, the linear optimization scheme might decrease the overall objective value at the expense of damaging one of the training objectives, leading to restricted solutions. In this paper, we rethink the optimization scheme for DA from a gradient-based perspective. We propose a Pareto Domain Adaptation (ParetoDA) approach to control the overall optimization direction, aiming to cooperatively optimize all training objectives. Specifically, to reach a desirable solution on the target domain, we design a surrogate loss mimicking target classification. To improve target-prediction accuracy to support the mimicking, we propose a target-prediction refining mechanism which exploits domain labels via Bayes’ theorem. On the other hand, since prior knowledge of weighting schemes for objectives is often unavailable to guide optimization to approach the optimal solution on the target domain, we propose a dynamic preference mechanism to dynamically guide our cooperative optimization by the gradient of the surrogate loss on a held-out unlabeled target dataset. Our theoretical analyses show that the held-out data can guide but will not be over-fitted by the optimization. Extensive experiments on image classification and semantic segmentation benchmarks demonstrate the effectiveness of ParetoDA

Deep domain adaptation methods have achieved appealing performance by learning transferable representations from a well-labeled source domain to a different but related unlabeled target domain. Most existing works assume source and target data share the identical label space, which is often difficult to be satisfied in many real-world applications. With the emergence of big data, there is a more practical scenario called partial domain adaptation, where we are always accessible to a more large-scale source domain while working on a relative small-scale target domain. In this case, the conventional domain adaptation assumption should be relaxed, and the target label space tends to be a subset of the source label space. Intuitively, reinforcing the positive effects of the most relevant source subclasses and reducing the negative impacts of irrelevant source subclasses are of vital importance to address partial domain adaptation challenge. This paper proposes an efficiently-implemented Deep Residual Correction Network (DRCN) by plugging one residual block into the source network along with the task-specific feature layer, which effectively enhances the adaptation from source to target and explicitly weakens the influence from the irrelevant source classes. Specifically, the plugged residual block, which consists of several fully-connected layers, could deepen basic network and boost its feature representation capability correspondingly. Moreover, we design a weighted class-wise domain alignment loss to couple two domains by matching the feature distributions of shared classes between source and target. Comprehensive experiments on partial, traditional and fine-grained cross-domain visual recognition demonstrate that DRCN is superior to the competitive deep domain adaptation approaches.

Vehicular crowdsensing (VCS) takes the advantage of vehicles' mobility and exploits both the crowd wisdom and sensing abilities offered by vehicle drivers' carried smart mobile devices and on-board sensors to accomplish challenging sensing tasks. The daily roadway commutes of vehicle drivers may form “virtual” mobile communities, called Vehicular Social Networks (VSNs). It offers an opportunity to include social network effect into incentive mechanism design where a driver can benefit from others' sensing strategy in one VSN. In this paper, we consider a non-cooperative VCS campaign where multiple vehicles are incentivized by dynamically priced tasks and social network effect. In order to maximize the overall utility of vehicle drivers, we propose a social-aware incentive mechanism by deep reinforcement learning (called DRL-SIM), to derive the optimal long term sensing strategy for all vehicles. Finally, numerical results are supplemented to show both the convergence and the effectiveness of DRL-SIM when compared with other baselines.

The core of many large-scale machine learning (ML) applications, such as neural networks (NN), support vector machine (SVM), and convolutional neural network (CNN), is the training algorithm that iteratively updates model parameters by processing massive datasets. From a plethora of studies aiming at accelerating ML, being data parallelization and parameter server, the prevalent assumption is that all data points are equivalently relevant to model parameter updating. In this article, we challenge this assumption by proposing a criterion to measure a data point's effect on model parameter updating, and experimentally demonstrate that the majority of data points are non-critical in the training process. We develop a slim learning framework, termed SlimML, which trains the ML models only on the critical data and thus significantly improves training performance. To such an end, SlimML efficiently leverages a small number of aggregated data points per iteration to approximate the criticalness of original input data instances. The proposed approach can be used by changing a few lines of code in a standard stochastic gradient descent (SGD) procedure, and we demonstrate experimentally, on NN regression, SVM classification, and CNN training, that for large datasets, it accelerates model training process by an average of 3.61 times while only incurring accuracy losses of 0.37 percent.

Mobile crowdsensing (MCS) represents a new sensing paradigm that utilizes the smart mobile devices to collect and share data. Traditional MCS systems mainly leverages the people carried smartphones and other wearable devices which are constrained by the limited sensing capability and battery power. With the popularity of unmanned vehicles like unmanned aerial vehicles (UAVs) and driverless cars, they can provide much more reliable, accurate and cost-efficient sensing services due to to their equipped more powerful sensors. In this paper, we propose a distributed control framework for energy-efficient and DIstributed VEhicle navigation with chaRging sTations, called “e-Divert”. It is a distributed multi-agent deep reinforcement learning (DRL) solution, which uses a convolutional neural network (CNN) to extract useful spatial features as the input to the actor-critic network to produce a real-time action. Also, e-Divert incorporates a distributed prioritized experience replay for better exploration and exploitation, and a long short-term memory (LSTM) enabled N-step temporal sequence modeling module. The solution fully explores the spatiotemporal nature of the considered scenario for better vehicle cooperation and competition between themselves and charging stations, to maximize the energy efficiency, data collection ratio, geographic fairness, and minimize the energy consumption simultaneously. Through extensive simulations, we find an appropriate set of hyperparameters that achieve the best performance, i.e., 5 actors in Ape-X architecture, priority exponent 0.5, and LSTM sequence length 3. Finally, we compare with four baselines including one state-of-the-art approach MADDPG. Results show that our proposed e-Divert significantly improves the energy efficiency, as compared to MADDPG, by 3.62 and 2.36 times on average when varying different numbers of vehicles and charging stations, respectively.

Deep learning (DL) models are increasingly built on federated edge participants holding local data. To enable insight extractions without the risk of information leakage, DL training is usually combined with differential privacy (DP). The core theme is to tradeoff learning accuracy by adding statistically calibrated noises, particularly to local gradients of edge learners, during model training. However, this privacy guarantee unfortunately degrades model accuracy due to edge learners' local noises, and the global noise aggregated at the central server. Existing DP frameworks for edge focus on local noise calibration via gradient clipping techniques, overlooking the heterogeneity and dynamic changes of local gradients, and their aggregated impact on accuracy. In this article, we present a systematical analysis that unveils the influential factors capable of mitigating local and aggregated noises, and design PrivateDL to leverage these factors in noise calibration so as to improve model accuracy while fulfilling privacy guarantee. PrivateDL features on: (i) sampling-based sensitivity estimation for local noise calibration and (ii) combining large batch sizes and critical data identification in global training. We implement PrivateDL on the popular Laplace/Gaussian DP mechanisms and demonstrate its effectiveness using Intel BigDL workloads, i.e., considerably improving model accuracy by up to 5X when comparing against existing DP frameworks.

With the exponential growth of data created at the network edge, decentralized and Gossip-based training of deep learning (DL) models on edge computing (EC) gains tremendous research momentum, owing to its capability to learn from resource-strenuous edge nodes with limited network connectivity. Today's edge devices are extremely heterogeneous, e.g., hardware and software stacks, and result in high performance variation of training time and inducing extra delay to synchronize and converge. The large body of prior art accelerates DL, being data or model parallelization, via a centralized server, e.g., parameter server scheme, which may easily turn into the system bottleneck or single point of failure. In this artice, we propose EdgeGossip, a framework specifically designed to accelerate the training process of decentralized and Gossip-based DL training for heterogeneous EC platforms. EdgeGossip features on: (i) low performance variation among multiple EC platforms during iterative training, and (ii) accuracy-aware training to fastly obtain best possible model accuracy. We implement EdgeGossip based on popular Gossip algorithms and demonstrate its effectiveness using real-world DL workloads, i.e., considerably reducing model training time by an average of 2.70 times while only incurring accuracy losses of 0.78 percent.

Tremendous research efforts have been made to thrive deep domain adaptation (DA) by seeking domain-invariant features. Most existing deep DA models only focus on aligning feature representations of task-specific layers across domains while integrating a totally shared convolutional architecture for source and target. However, we argue that such strongly-shared convolutional layers might be harmful for domain-specific feature learning when source and target data distribution differs to a large extent. In this paper, we relax a shared-convnets assumption made by previous DA methods and propose a Domain Conditioned Adaptation Network (DCAN), which aims to excite distinct convolutional channels with a domain conditioned channel attention mechanism. As a result, the critical low-level domain-dependent knowledge could be explored appropriately. As far as we know, this is the first work to explore the domain-wise convolutional channel activation for deep DA networks. Moreover, to effectively align high-level feature distributions across two domains, we further deploy domain conditioned feature correction blocks after task-specific layers, which will explicitly correct the domain discrepancy. Extensive experiments on three cross-domain benchmarks demonstrate the proposed approach outperforms existing methods by a large margin, especially on very tough cross-domain learning tasks.

Spatial crowdsourcing (SC) utilizes the potential of a crowd to accomplish certain location based tasks. Although worker scheduling has been well studied recently, most existing works only focus on the static deployment of workers but ignore their temporal movement continuity. In this paper, we explicitly consider the use of unmanned vehicular workers, e.g., drones and driverless cars, which are more controllable and can be deployed in remote or dangerous areas to carry on long-term and hash tasks as a vehicular crowdsourcing (VC) campaign. We propose a novel deep reinforcement learning (DRL) approach for curiosity-driven energy-efficient worker scheduling, called "DRL-CEWS", to achieve an optimal trade-off between maximizing the collected amount of data and coverage fairness, and minimizing the overall energy consumption of workers. Specifically, we first utilize a chief-employee distributed computational architecture to stabilize and facilitate the training process. Then, we propose a spatial curiosity model with a sparse reward mechanism to help derive the optimal policy in large crowdsensing space with unevenly distributed data. Extensive simulation results show that DRL-CEWS outperforms the state-of-the-art methods and baselines, and we also visualize the benefits curiosity model brings and show the impact of two hyperparameters.

With the popularity of drones and driverless cars, vehicular crowdsensing (VCS) becomes increasingly widely-used by taking advantage of their high-precision sensors and durability in harsh environments. Since abrupt sensing tasks usually cannot be prepared beforehand, we need a generic control logic fit-for-use all tasks which are similar in nature, but different in their own settings like Point-of-Interest (PoI) distributions. The objectives include to simultaneously maximize the data collection amount, geographic fairness, and minimize the energy consumption of all vehicles for all tasks, which usually cannot be explicitly expressed in a closed-form equation, thus not tractable as an optimization problem. In this paper, we propose a deep reinforcement learning (DRL)-based centralized control, distributed execution framework for multi-task-oriented VCS, called "DRL-MTVCS". It includes an asynchronous architecture with spatiotemporal state information modeling, multi-task-oriented value estimates by adaptive normalization, and auxiliary vehicle action exploration by pixel control. We compare with three baselines, and results show that DRL-MTVCS outperforms all others in terms of energy efficiency when varying different numbers of tasks, vehicles, charging stations and sensing range.

Different from using human-centric mobile devices like smartphones, unmanned aerial vehicles (UAVs) can be utilized to form a new UAV crowdsensing paradigm, where UAVs are equipped with build-in high-precision sensors, to provide data collection services especially for emergency situations like earthquakes or flooding. In this paper, we aim to propose a new deep learning based framework to tackle the problem that a group of UAVs energy-efficiently and cooperatively collect data from low-level sensors, while charging the battery from multiple randomly deployed charging stations. Specifically, we propose a new deep model called "j-PPO+ConvNTM" which contains a novel spatiotemporal module "Convolution Neural Turing Machine" (ConvNTM) to better model long-sequence spatiotemporal data, and a deep reinforcement learning (DRL) model called "j-PPO", where it has the capability to make continuous (i.e., route planing) and discrete (i.e., either to collect data or go for charging) action decisions simultaneously for all UAVs. Finally, we perform extensive simulation to show its illustrative movement trajectories, hyperparameter tuning, ablation study, and compare with four other baselines.

Heterogeneous domain adaptation (HDA) transfers knowledge across source and target domains that present heterogeneities e.g., distinct domain distributions and difference in feature type or dimension. Most previous HDA methods tackle this problem through learning a domain-invariant feature subspace to reduce the discrepancy between domains. However, the intrinsic semantic properties contained in data are under-explored in such alignment strategy, which is also indispensable to achieve promising adaptability. In this paper, we propose a Simultaneous Semantic Alignment Network (SSAN) to simultaneously exploit correlations among categories and align the centroids for each category across domains. In particular, we propose an implicit semantic correlation loss to transfer the correlation knowledge of source categorical prediction distributions to target domain. Meanwhile, by leveraging target pseudo-labels, a robust triplet-centroid alignment mechanism is explicitly applied to align feature representations for each category. Notably, a pseudo-label refinement procedure with geometric similarity involved is introduced to enhance the target pseudo-label assignment accuracy. Comprehensive experiments on various HDA tasks across text-to-image, image-to-image and text-to-text successfully validate the superiority of our SSAN against state-of-the-art HDA methods. The code is publicly available at https://github.com/BIT-DA/SSAN.

Mobile crowdsensing (MCS) is an attractive and innovative paradigm in which a crowd of users equipped with smart mobile devices (such as smartphones and iPads), and more recently unmanned vehicles (UVs, e.g., driverless cars and drones) conduct sensing tasks in mobile social networks by fully exploiting their carried diverse embedded sensors. These devices, especially UVs, are usually constrained by limited sensing range and energy reserve of devices, which contribute to the restriction of one single UV task performance, and thus UV collaborations are fully favored. In this article, we explicitly consider navigating a group of UVs to collect different kinds of data in a city, with the presence of multiple charging stations. Different from the existing approaches that solve the problem by forming a constrained optimization problem, we propose a novel sequential deep model called “PPO+LSTM,” which contains a sequential model LSTM and is trained with proximal policy optimization (PPO), for assigning tasks and planning route. We evaluate our model in different network settings when comparing with other state-of-the-art solutions, and we also show the impact of important hyperparameters of our model. Results show that our solution outperforms all others in terms of energy efficiency, data collection ratio, and geographic fairness.

Modern distributed engines are increasingly deployed to accelerate large-scaled deep learning (DL) training jobs. While the parallelism of distributed workers/nodes promises the scalability, the computation and communication overheads of the underlying iterative solving algorithms, e.g., stochastic gradient decent, unfortunately become the bottleneck for distributed DL training jobs. Existing approaches address such limitations by designing more efficient synchronization algorithms and model compressing techniques, but do not adequately address issues relating to processing massive datasets. In this article, we propose ClipDL, which accelerates the deep learning systems by simultaneously decreasing the number of model parameters as well as reducing the computations on critical data only. The core component of ClipDL is the estimation of critical set based on the observation that large proportions of input data have little influence on model parameter updating in many prevalent DL algorithms. We implemented ClipDL on Spark (a popular distributed engine for big data) and BigDL (based on de-factor distributed DL training architecture, parameter server), and integrated it with representative model compression techniques. The exhaustive experiments on real DL applications and datasets show ClipDL accelerates model training process by an average of 2.32 times while only incurring accuracy losses of 1.86 percent.

In this paper, we aim to design a fully-distributed control solution to navigate a group of unmanned aerial vehicles (UAVs), as the mobile Base Stations (BSs) to fly around a target area, to provide long-term communication coverage for the ground mobile users. Different from existing solutions that mainly solve the problem from optimization perspectives, we proposed a decentralized deep reinforcement learning (DRL) based framework to control each UAV in a distributed manner. Our goal is to maximize the temporal average coverage score achieved by all UAVs in a task, maximize the geographical fairness of all considered point-of-interests (PoIs), and minimize the total energy consumptions, while keeping them connected and not flying out of the area border. We designed the state, observation, action space, and reward in an explicit manner, and model each UAV by deep neural networks (DNNs). We conducted extensive simulations and found the appropriate set of hyperparameters, including experience replay buffer size, number of neural units for two fully-connected hidden layers of actor, critic, and their target networks, and the discount factor for remembering the future reward. The simulation results justified the superiority of the proposed model over the state-of-the-art DRL-EC3 approach based on deep deterministic policy gradient (DDPG), and three other baselines.

The explosive increase of mobile devices with built-in sensors such as GPS, accelerometer, gyroscope and camera has made the design of mobile crowdsensing (MCS) applications possible, which create a new interface between humans and their surroundings. Until now, various MCS applications have been designed, where the task initiators (TIs) recruit mobile users (MUs) to complete the required sensing tasks. In this paper, deep reinforcement learning (DRL) based techniques are investigated to address the problem of assigning satisfactory but profitable amount of incentives to multiple TIs and MUs as a MCS game. Specifically, we first formulate the problem as a multi-leader and multi-follower Stackelberg game, where TIs are the leaders and MUs are the followers. Then, the existence of the Stackelberg Equilibrium (SE) is proved. Considering the challenge to compute the SE, a DRL based Dynamic Incentive Mechanism (DDIM) is proposed. It enables the TIs to learn the optimal pricing strategies directly from game experiences without knowing the private information of MUs. Finally, numerical experiments are provided to illustrate the effectiveness of the proposed incentive mechanism compared with both state-of-the-art and baseline approaches.

Visual domain adaptation aims to seek an effective transferable model for unlabeled target images by benefiting from the well-labeled source images following different distributions. Many recent efforts focus on extracting domain-invariant image representations via exploring target pseudo labels, predicted by the source classifier, to further mitigate the conditional distribution shift across domains. However, two essential factors are overlooked by most existing methods: 1) the learned transferable features should be not only domain invariant but also category discriminative; and 2) the target pseudo label is a two-edged sword to cross-domain alignment. In other words, the wrongly predicted target labels may hinder the class-wise domain matching. In this article, to address these two issues simultaneously, we propose a discriminative transfer feature and label consistency (DTLC) approach for visual domain adaptation problems, which can naturally unify cross-domain alignment with discriminative information preserved and label consistency of source and target data into one framework. To be specific, DTLC first incorporates class discriminative information by penalizing the maximum distance of data pair in the same class and the minimum distance of data pair sharing the different labels for each data into the distribution alignment of both domains. The target pseudo labels are then refined based on the label consistency within the domains. Thus, the transfer feature learning and coarse-to-fine target labels would be coupled to benefit each other in an iterative way. Comprehensive experiments on several visual cross-domain benchmarks verify that DTLC can gain remarkable margins over state-of-the-art (SOTA) nondeep visual domain adaptation methods and even be comparable to competitive deep domain adaptation ones.

Domain adaptation aims to transfer the enriched label knowledge from large amounts of source data to unlabeled target data. It has raised significant interest in multimedia analysis. Existing researches mainly focus on learning domain-wise transferable representations via statistical moment matching or adversarial adaptation techniques, while ignoring the class-wise mismatch across domains, resulting in inaccurate distribution alignment. To address this issue, we propose a Joint Adversarial Domain Adaptation (JADA) approach to simultaneously align domain-wise and class-wise distributions across source and target in a unified adversarial learning process. Specifically, JADA attempts to solve two complementary minimax problems jointly. The feature generator aims to not only fool the well-trained domain discriminator to learn domain-invariant features, but also minimize the disagreement between two distinct task-specific classifiers' predictions to synthesize target features near the support of source class-wisely. As a result, the learned transferable features will be equipped with more discriminative structures, and effectively avoid mode collapse. Additionally, JADA enables an efficient end-to-end training manner via a simple back-propagation scheme. Extensive experiments on several real-world cross-domain benchmarks, including VisDA-2017, ImageCLEF, Office-31 and digits, verify that JADA can gain remarkable improvements over other state-of-the-art deep domain adaptation approaches.

Actions always refer to complex vision variations in a long-range redundant video sequence. Instead of focusing on limited range sequence, i.e. convolution on adjacent frames, in this paper, we proposed an action recognition approach with bootstrapping based long-range temporal context attention. Specifically, due to vision variations of the local region across frames, we target at capturing temporal context by proposing the Temporal Pixels based Parallel-head Attention (TPPA) block. In TPPA, we apply the self-attention mechanism between local regions at the same position across temporal frames to capture the interaction impacts. Meanwhile, to deal with video redundancy and capture long-range context, the TPPA is extended to the Random Frames based Bootstrapping Attention (RFBA) framework. While the bootstrapping sampling frames have the same distribution of the whole video sequence, the RFBA not only captures longer temporal context with only a few sampling frames but also has comprehensive representation through multiple sampling. Furthermore, we also try to apply this temporal context attention to image-based action recognition, by transforming the image into "pseudo video" with the spatial shift. Finally, we conduct extensive experiments and empirical evaluations on two most popular datasets:UCF101 for videos andStanford40 for images. In particular, our approach achieves top-1 accuracy of $91.7%$ in UCF101 and mAP of $90.9%$ in Stanford40.

High-quality data collection is crucial for mobile crowd sensing (MCS) with various applications like smart cities and emergency rescues, where various unmanned mobile terminals (MTs), e.g., driverless cars and unmanned aerial vehicles (UAVs), are equipped with different sensors that aid to collect data. However, they are limited with fixed carrying capacity, and thus, MT's energy resource and sensing range are constrained. It is quite challenging to navigate a group of MTs to move around a target area to maximize their total amount of collected data with the limited energy reserve, while geographical fairness among those point-of-interests (PoIs) should also be maximized. It is even more challenging if fully distributed execution is enforced, where no central control is allowed at the backend. To this end, we propose to leverage emerging deep reinforcement learning (DRL) techniques for directing MT's sensing and movement and to present a novel and highly efficient control algorithm, called energy-efficient distributed MCS (Edics). The proposed neural network integrates convolutional neural network (CNN) for feature extraction and then makes decision under the guidance of multi-agent deep deterministic policy gradient (DDPG) method in a fully distributed manner. We also propose two enhancements into Edics with N-step return and prioritized experienced replay buffer. Finally, we evaluate Edics through extensive simulations and found the appropriate set of hyperparameters in terms of number of CNN hidden layers and neural units for all the fully connected layers. Compared with three commonly used baselines, results have shown its benefits.

With the rapid development of smart mobile terminals (MTs), various industrial Internet of things (IIoT) applications can fully leverage them to collect and share data for providing certain services. However, two key challenges still remain. One is how to achieve high-quality data collection with limited MT energy resource and sensing range. Another is how to ensure security when sharing and exchanging data among MTs, to prevent possible device failure, network communication failure, malicious users or attackers, etc. To this end, we propose a blockchain-enabled efficient data collection and secure sharing scheme combining Ethereum blockchain and deep reinforcement learning (DRL) to create a reliable and safe environment. In this scheme, DRL is used to achieve the maximum amount of collected data, and the blockchain technology is used to ensure security and reliability of data sharing. Extensive simulation results demonstrate that the proposed scheme can provide higher security level and stronger resistance to attack than a traditional database based data sharing scheme for different levels/types of attacks.

In this paper, we propose to leverage the emerging deep learning techniques for sequential modeling of user interests based on big social data, which takes into account influence of their social circles. First, we present a preliminary analysis for two popular big datasets from Yelp and Epinions. We show statistically sequential actions of all users and their friends, and discover both temporal autocorrelation and social influence on decision making, which motivates our design. Then, we present a novel hybrid deep learning model, Social-Aware Long Short-Term Memory (SA-LSTM), for predicting the types of item/PoIs that a user will likely buy/visit next, which features stacked LSTMs for sequential modeling and an autoencoder-based deep model for social influence modeling. Moreover, we show that SA-LSTM supports end-to-end training. We conducted extensive experiments for performance evaluation using the two real datasets from Yelp and Epinions. The experimental results show that (1) the proposed deep model significantly improves prediction accuracy compared to widely used baseline methods; (2) the proposed social influence model works effectively; and (3) going deep does help improve prediction accuracy but a not-so-deep deep structure leads to the best performance.

Mobile crowdsensing (MCS) is a new paradigm to collect sensing data and infer useful knowledge over a vast area for numerous monitoring applications. In urban environments, as more and more applications need to utilize multi-source sensing information, it is almost indispensable to develop a generic mechanism supporting multiple concurrent MCS task assignment. However, most existing multi-task assignment methods focus on homogeneous tasks. Due to the diverse spatiotemporal task requirements and sensing contexts, MCS tasks often differ from each other in many aspects (e.g., spatial coverage, temporal interval). To this end, in the paper, we present and formalize an important Heterogeneous Multi-Task Assignment (HMTA) problem in mobile crowdsensing systems, and try to maximize data quality and minimize total incentive budget. By leveraging the implicit spatiotemporal correlations among heterogeneous tasks, we propose a two-stage HMTA problem-solving approach to effectively handle multiple concurrent tasks in a shared resource pool. Finally, in order to improve the assignment search efficiency, a decomposition-and-combination framework is devised to accommodate large-scale problem scenario. We evaluate our approach extensively using two large-scale real-world data sets. The experimental results validate the effectiveness and efficiency of our proposed approach.

Mobile crowd sensing is a new paradigm that enables smart mobile devices to collect and share various types of sensing data in urban environments. However, new challenges arise: one is how to evaluate the quality of data each mobile user potentially is capable of providing; another is how to allocate a satisfactory yet profitable amount of reward to mobile users in order to keep them participating in crowd sensing tasks. In this paper, we first introduce a mathematical model for characterizing quality of sensing data to be contributed by mobile users. Then, we present a utility function and formulate an optimization problem for the platform, who recruits participants to contribute sensing data, to maximize the amount of high quality sensing data under a limited task budget. We next present an effective and quality-aware incentive mechanism to solve this problem for online scenarios where participants may arrive or leave at any random time. Moreover, the proposed incentive mechanism allows the platform to provide selected participants with an extra bonus according to task completion level and their previous performance to motivate them further. We formally show the proposed mechanism has the desirable properties of truthfulness, individual rationality, budgetary feasibility, and computational efficiency. We compare the proposed scheme with existing methods via simulation using a real dataset. Extensive simulation results well justify the effectiveness and robustness of the proposed approach, e.g, compared with another online method “OMG”, the gap to the optimum for our proposed Online-QIM approach is reduce by 33.3 percent when budget B = 1000.

Cluster schedulers provide flexible resource sharing mechanism for best-effort cloud jobs, which occupy a majority in modern datacenters. Properly tuning a scheduler's configurations is the key to these jobs' performance because it decides how to allocate resources among them. Today's cloud scheduling systems usually rely on cluster operators to set the configuration and thus overlook the potential performance improvement through optimally configuring the scheduler according to the heterogeneous and dynamic cloud workloads. In this paper, we introduce AdaptiveConfig, a run-time configurator for cluster schedulers that automatically adapts to the changing workload and resource status in two steps. First, a comparison approach estimates jobs' performances under different configurations and diverse scheduling scenarios. The key idea here is to transform a scheduler's resource allocation mechanism and their variable influence factors (configurations, scheduling constraints, available resources, and workload status) into business rules and facts in a rule engine, thereby reasoning about these correlated factors in job performance comparison. Second, a workload-adaptive optimizer transforms the cluster-level searching of huge configuration space into an equivalent dynamic programming problem that can be efficiently solved at scale. We implement AdaptiveConfig on the popular YARN Capacity and Fair schedulers and demonstrate its effectiveness using real-world Facebook and Google workloads, i.e., successfully finding best configurations for most of scheduling scenarios and considerably reducing latencies by a factor of two with low optimization time.

Modern communication networks have become very complicated and highly dynamic, which makes them hard to model, predict and control. In this paper, we develop a novel experience-driven approach that can learn to well control a communication network from its own experience rather than an accurate mathematical model, just as a human learns a new skill (such as driving, swimming, etc). Specifically, we, for the first time, propose to leverage emerging Deep Reinforcement Learning (DRL) for enabling model-free control in communication networks; and present a novel and highly effective DRL-based control framework, DRL-TE, for a fundamental networking problem: Traffic Engineering (TE). The proposed framework maximizes a widely-used utility function by jointly learning network environment and its dynamics, and making decisions under the guidance of powerful Deep Neural Networks (DNNs). We propose two new techniques, TE-aware exploration and actor-critic-based prioritized experience replay, to optimize the general DRL framework particularly for TE. To validate and evaluate the proposed framework, we implemented it in ns-3, and tested it comprehensively with both representative and randomly generated network topologies. Extensive packet-level simulation results show that 1) compared to several widely-used baseline methods, DRL-TE significantly reduces end-to-end delay and consistently improves the network utility, while offering better or comparable throughput; 2) DRL-TE is robust to network changes; and 3) DRL-TE consistently outperforms a state-of-the-art DRL method (for continuous control), Deep Deterministic Policy Gradient (DDPG), which, however, does not offer satisfying performance.

Unmanned aerial vehicles (UAVs) can be used to serve as aerial base stations to enhance both the coverage and performance of communication networks in various scenarios, such as emergency communications and network access for remote areas. Mobile UAVs can establish communication links for ground users to deliver packets. However, UAVs have limited communication ranges and energy resources. Particularly, for a large region, they cannot cover the entire area all the time or keep flying for a long time. It is thus challenging to control a group of UAVs to achieve certain communication coverage in a long run, while preserving their connectivity and minimizing their energy consumption. Toward this end, we propose to leverage emerging deep reinforcement learning (DRL) for UAV control and present a novel and highly energy-efficient DRL-based method, which we call DRL-based energy-efficient control for coverage and connectivity (DRL-EC3). The proposed method 1) maximizes a novel energy efficiency function with joint consideration for communications coverage, fairness, energy consumption and connectivity; 2) learns the environment and its dynamics; and 3) makes decisions under the guidance of two powerful deep neural networks. We conduct extensive simulations for performance evaluation. Simulation results have shown that DRL-EC3significantly and consistently outperform two commonly used baseline methods in terms of coverage, fairness, and energy consumption.

Mobile crowdsourcing (MCS) is now an important source of information for smart cities, especially with the help of unmanned aerial vehicles (UAVs) and driverless cars. They are equipped with different kinds of high-precision sensors, and can be scheduled/controlled completely during data collection, which will make MCS system more robust. However, they are limited to energy constraint, especially for long-term, long-distance sensing tasks, and cities are almost too crowded to set stationary charging station. Towards this end, in this paper we propose to leverage emerging deep reinforcement learning (DRL) techniques for enabling model-free unmanned vehicles control, and present a novel and highly effective control framework, called “DRL-RVC.” It utilizes the powerful convolutional neural network for feature extraction of the necessary information (including sample distribution, traffic flow, etc.), then makes decisions under the guidance of the deep Q network. That is, UAVs will cruise in the city without control and collect most required data in the sensing region, while mobile unmanned charging station will reach the charging point in the shortest possible time. Finally, we validate and evaluate the proposed framework via extensive simulations based on a real dataset in Rome. Extensive simulation results well justify the effectiveness and robustness of our approach.

A signature is a useful human feature in our society, and determining the genuineness of a signature is very important. A signature image is typically analyzed for its genuineness classification; however, increasing classification accuracy while decreasing computation time is difficult. Many factors affect image quality and the genuineness classification, such as contour damage and light distortion or the classification algorithm. To this end, we propose a mobile computing method of signature image authentication (SIA) with improved recognition accuracy and reduced computation time. We demonstrate theoretically and experimentally that the proposed golden global‐local (G‐L) algorithm has the best filtering result compared with the methods of mean filtering, medium filtering, and Gaussian filtering. The developed minimum probability threshold (MPT) algorithm produces the best segmentation result with minimum error compared with methods of maximum entropy and iterative segmentation. In addition, the designed convolutional neural network (CNN) solves the light distortion problem for detailed frame feature extraction of a signature image. Finally, the proposed SIA algorithm achieves the best signature authentication accuracy compared with CNN and sparse representation, and computation times are competitive. Thus, the proposed SIA algorithm can be easily implemented in a mobile phone.

Mobile Crowd sensing is a new paradigm that encourages ordinary people to collect and share sensing data with their smart devices. However, the uncontrollable data quality is one of the critical problems that is potentially harmful to guarantee the availability and preciseness of mobile crowd sensing based services. In this paper, we propose a quality aware data collection mechanism based on a realistic scenario where participants arrive and report their conditions sequentially one by one. When a participant arrives, the mechanism first forecasts the amount of high quality data he#x002F;she may contribute by employing the expectation of Binomial-Poisson distribution, and then a two-level iterative algorithm is employed to calculate its parametric values. After, our designed mechanism decides to select the participant or not by combining with his#x002F;her requested reward. Extensive simulation results well justify the effectiveness and robustness of our approach, compared with another schemes.

Participatory Crowdsourcing System (PCS) is now an important source of information for big data applications. In PCS, required environmental parameters are collected and shared by ordinary citizens. And the limited range of human activities and limited power or capability of smart devices make the PCS facing a lot of problems. But the wide use of robots, such as Unmanned Aerial Vehicle (UAV), can effectively alleviate the current problems in PCS. Lots of precision sensors can be installed into the robots, which makes sensory data collected by robots more trustful. However, the power and data transmission capability of robots are also limited. Meanwhile, it is much more expensive to use robots than recruit human participants. Thus, it will be most useful and necessary to use robots in PCS, let the robots and human participants working together, rather than using robots to replace the human participants. Obviously, the cooperation of robots and human participants will also make PCS facing new challenges, such as how to use the least amount of robots to reduce costs, etc.. In this paper, we analyze the different application scenarios of robots-aided PCS, and propose possible solutions to the existing problems. Finally, we discuss possible future work for robots-aided PCS.

Recognizing and retrieving multimedia content with movie/TV series actors, especially querying actor-specific videos in large scale video datasets, has attracted much attention in both the video processing and computer vision research field. However, many existing methods have low efficiency both in training and testing processes and also a less than satisfactory performance. Considering these challenges, in this paper, we propose an efficient cloud-based actor identification approach with batch-orthogonal local-sensitive hashing (BOLSH) and multi-task joint sparse representation classification. Our approach is featured by the following: 1) videos from movie/TV series are segmented into shots with the cloud-based shot boundary detection; 2) while faces in each shot are detected and tracked, the cloud-based BOLSH is then implemented on these faces for feature description; 3) the sparse representation is then adopted for actor identification in each shot; and 4) finally, a simple application, actor-specific shots retrieval is realized to verify our approach. We conduct extensive experiments and empirical evaluations on a large scale dataset, to demonstrate the satisfying performance of our approach considering both accuracy and efficiency.

Recommender systems have changed the way people originally find products, information, and even their social circles. However, most existing research activities neglect its time-varying feature, i.e., the growing input data, the change of user behaviors. In order to sustain the high accuracy of recommendations, systems have to be updated regularly. However, the more often the update proceeds, the more cost of time and other computational resources. Thus, it is critical to strike the balance between accuracy and cost. In this paper, we propose an adaptive recommender system by using feedback control frameworks. The proposed solution continuously monitors its changes and estimates the loss of performance (in terms of accuracy) from two perspectives: data problem(data aging and data deficient) in training set, and changes of user behavior by “revisiting ratio”. When the benefit of performing an update exceeds the cost of resources, the system update itself. Theoretical analysis and extensive results by using a real data set are supplemented to show the advantages of the proposed system.

Dynamic event detection by using participatory sensing paradigms has received growing interests in recent years, where detection tasks are assigned to smart device users who can potentially collect needed sensory data from the equipped sensors. These data can be utilized to detect interested events like noise, air pollution, or even earthquake. Since most existing solutions focus on centralized detection approaches that, however, usually cause heavy communication overhead, it is strongly desired to design distributed solutions to reduce energy consumption while achieving a high level of detection accuracy. In this paper, we first present a novel Minimum Cut based centralized detection algorithm as the performance benchmark, and then introduce a novel distributed, energy-efficient solution, where an optimization problem is formulated and an optimal solution is derived. Simulations based on a real-trace driven data set in Beijing demonstrate the effectiveness of our proposed algorithms.

Crowdsourcing systems, by using smart devices like smartphones and iPad, have been widely used in various domains, but are currently facing new challenges. On one hand, different tasks offer different amount of incentive budgets in multitask systems, thus those tasks pay more should be satisfied preferentially. On the other hand, even if there are not enough budget for a sensing task, the system should also try to provide as much sensory data as possible to obtain potential clients. Furthermore, it is challenging to select an “optimal” set of participants as data contributors due to the above two points. In this paper, first, we introduce a metric to describe how the collected sensory data can quantify task's the multi-dimensional data requirements, in terms of data distribution in spatiotemporal domains. Second, we propose a “task priority model” based on incentive budget, to explicitly quantify the relationship between the incentive budget usage and task priority. Then, we present a quality of information (QoI) aware participant selection approach as a suboptimal solution to the defined optimization problem. Finally, we compare our proposed scheme with existing methods via extensive simulations based on the real movement traces of ordinary citizens in Beijing. Extensive simulation results well justify the effectiveness and robustness of our approach.

This paper proposes a light-weight online classification method to detect smarthpone user's postural actions, such as sitting, standing, walking, and running. These actions are named as “user states” since they are inferred after the analysis of data acquired from the smartphones equipped accelerometer sensors. To differentiate one user state from another, many studies can be found in the literature. However, this study differs from all others by offering a computational lightweight and online classification method without knowing any priori information. Moreover, the proposed method not only provides a standalone solution in differentiation of user states, but also it assists other widely used offline supervised classification methods by automatically generating training data classes and/or input system matrices. Furthermore, we improve these existing methods for the purpose of online processing by reducing the required computational burden. Extensive experimental results show that the proposed method makes a solid differentiation in user states even when the sensor is being operated under slower sampling frequencies.

The Internet of Things (IoT) envisions billions of sensors deployed around us and connected to the Internet, where the mobile crowd sensing technologies are widely used to collect data in different contexts of the IoT paradigm. Due to the popularity of Big Data technologies, processing and storing large volumes of data have become easier than ever. However, large-scale data management tasks still require significant amounts of resources that can be expensive regardless of whether they are purchased or rented (e.g., pay-as-you-go infrastructure). Further, not everyone is interested in such large-scale data collection and analysis. More importantly, not everyone has the financial and computational resources to deal with such large volumes of data. Therefore, a timely need exists for a cloud-integrated mobile crowd sensing platform that is capable of capturing sensors data, on-demand, based on conditions enforced by the data consumers. In this paper, we propose a context-aware, specifically, location and activity-aware mobile sensing platform called context-aware mobile sensor data engine (C-MOSDEN) for the IoT domain. We evaluated the proposed platform using three real-world scenarios that highlight the importance of selective sensing. The computational effectiveness and efficiency of the proposed platform are investigated and are used to highlight the advantages of context-aware selective sensing.

Participatory sensing systems can be used for concurrent event monitoring applications, like noise levels, fire, and pollutant concentrations. However, they are facing new challenges as to how to accurately detect the exact boundaries of these events, and further, to select the most appropriate participants to collect the sensing data. On the one hand, participants’ handheld smart devices are constrained with different energy conditions and sensing capabilities, and they move around with uncontrollable mobility patterns in their daily life. On the other hand, these sensing tasks are within time-varying quality-of-information (QoI) requirements and budget to afford the users’ incentive expectations. Toward this end, this article proposes an event-driven QoI-aware participatory sensing framework with energy and budget constraints. The main method of this framework is event boundary detection. For the former, a two-step heuristic solution is proposed where the coarse-grained detection step finds its approximation and the fine-grained detection step identifies the exact location. Participants are selected by explicitly considering their mobility pattern, required QoI of multiple tasks, and users’ incentive requirements, under the constraint of an aggregated task budget. Extensive experimental results, based on a real trace in Beijing, show the effectiveness and robustness of our approach, while comparing with existing schemes.

Roadside units (RSUs) are public and personal wireless access points that can provide communications with infrastructure in ad hoc vehicular networks. We present CLOCS (Counting and Localization using Online Compressive Sensing), a novel system to retrieve both the number and locations of RSUs through war-driving. CLOCS employs online compressive sensing (CS), where received signal strength (RSS) values are recorded at runtime, and the number and location of RSUs are recovered immediately based on limited RSS readings. CLOCS also uses fine retrieval based on an expectation maximization method along the driving route. Extensive simulation results and experiments in a real testbed deployed in the campus of the University of California, Irvine, confirm that CLOCS can successfully reduce the number of measurements for RSU recovery, while maintaining satisfactory counting and localization accuracy. In addition, data dissemination, time cost, and effects of different mobile scenarios using CLOCS are analyzed, and the impact of CLOCS on network connectivity is studied using Microsoft VanLan traces.

Software Defined Networking (SDN) is an emerging networking paradigm that assumes a logically centralized control plane separated from the data plane. Despite all its advantages, separating the control and data planes introduces new challenges regarding resilient communications between the two. That is, disconnections between switches and their controllers could result in substantial packet loss and performance degradation. To achieve resilient control traffic forwarding, this paper investigates the protection of control traffic in SDNs with multiple controllers. We propose a control traffic protection scheme that combines both local rerouting and constrained reverse forwarding protections. This scheme enables switches to locally react to failures and redirect the control traffic to controllers by using standby backup forwarding options. Our goal is then to find a set of primary routes for control traffic, called protection control network, where as much control traffic as possible can benefit from the proposed protection scheme. We formulate the protection control network problem and develop an algorithm to solve it. Simulation results on real topologies show that our approach significantly improves the resilience of control traffic.

In increasingly popular participatory sensing systems, new challenges are arising to select the most appropriate participants when considering their hand-held smart device's different energy conditions, uncontrollable mobility pattern, and associated sensing capabilities to best satisfy the quality-of-information (QoI) requirements of sensing tasks. This paper proposes a QoI-aware energy-efficient participant selection strategy, where four key design elements are proposed. First is QoI satisfaction metric of a sensing task that uses the data granularity and quantity collected by participants to measure to what extend the task's QoI requirements are satisfied. Second is an “energy consumption index”, which estimates the impact of energy cost during the data collection on different participant's smart devices with different remaining energy levels. Third is the estimation of the collected amount of data by participants, where a probability-based movement model is proposed. Fourth is the proposal of a multi-objective constrained optimization problem for participant selection, where task QoI requirements and energy consumption index of all participants are taken as optimization objectives, and solved by our proposed suboptimal, easy-to-implement solution. Real and extensive trace-based experiments show that, the proposed participant selection scheme can well balance the trade-off between the task QoI and energy consumptions by selecting most efficient participants, compared with existing schemes.

Admission control (AC) is highly important in packet networks with quality-of-service (QoS) requirements since the uncontrolled admission of new data connections can jeopardize the QoS of existing connections and degrade the overall network performance. It requires a priory knowledge of the network capacity, the estimation of which is intricate and complex due to the operational characteristics of various communication protocols, complex network topologies, and dynamic traffic behavior with QoS requirements. In this work, we propose a generic admission control (GAC) methodology where the network conditions between any given ingress-egress node pair are summarized into a single parameter, referred to as the "QoS index". It accounts for various traffic volumes and QoS requirements, like throughput, delay, and packet error rate. Using this QoS index we track the network performance by predicting the potential impact of a new connection admission on the index. The decision depends on whether the predicted value lies below 1. We discuss its wide applicability and feasibility to many types of packet networks, wired or wireless, independent of what communication protocols in use at various layers. We finally validate the proposed GAC methodology by extensive simulations, confirming a significant improvement in terms of network goodput and QoS outage probability compared to other existing statistics-based AC methodologies.

Cooperative communication has been proven effective in enhancing the performance of wireless networks. In this paper, we propose an adaptive multi-relay selection with power allocation mechanism to offer energy fairness at each node for a cooperative network. Unlike traditional approaches where all nodes are considered to transmit in a collaborative manner, we explicitly consider the situation where each node exhibits some degree of selfishness behavior. By introducing novel concepts of the selfishness index and utility function which denote node's benefit from cooperation, we show that the proposed cooperation scheme can well balance the energy consumption among nodes as well as maximize the network lifetime. Theoretical analysis and extensive simulation results are supplemented to show advantages of the proposed approach in power saving and power allocation fairness.

In Opportunistic Mobile Networks (OppNets), a large amount of energy is consumed by idle listening, instead of infrequent data exchange. This makes energy saving a challenging and fundamental problem in OppNets, since nodes are typically battery-powered. Asynchronous duty-cycle operation is a promising approach for energy saving in OppNets, however, if its working schedule is not effectively designed, it may also cause significant network performance degradation. Therefore, it is pressing to design an energy-efficient working schedule for duty-cycle OppNets. In this paper, we first analyze the contact process in duty-cycle OppNets, then propose an adaptive working schedule for duty-cycle OppNets. The proposed schedule uses the past recorded contact histories to predict the future contact information, so as to adaptively configure the working schedule of each node in the network. Finally, extensive real trace-driven simulations are conducted to evaluate the performance of our proposed schedule. The results show that our proposed adaptive working schedule is superior to the random working schedule in terms of delivery ratio and delivery delay.

Mobile device based sensing provides a vast variety of user contextual information. To continuously acquire, classify and reason any context through built-in sensors cause mobile device batteries to deplete rapidly. Therefore, an urgent need has arisen to apply power saving approaches at the low level sensor operations while inferring user relevant context. In this paper, an adaptive sensing strategy is proposed to fill the mentioned deficiency within the realm of Human Activity Recognition (HAR). The strategy aims at inferring any user relevant postural activities through smartphone accelerometer by providing two intuitive power saving methods, which assign adaptive pairs of sampling frequencies and duty cycles during sensor operations. The proposed methods show from 20% up to 50% efficiency enhancement in power consumption caused by the accelerometer with respect to overall 15% accuracy decrease in context inference depending on the degree of user activity dynamics.

Considering physical sensors with certain sensing capabilities in an Internet-of-Things (IoT) sensory environment, in this paper, we propose an efficient energy management framework to control the duty cycles of these sensors under quality-of-information (QoI) experience in a multi-task-oriented IoT sensory environment. Contrary to past research efforts, our proposal is transparent and compatible both with the underlying low-layer protocols and diverse applications, and preserving energy-efficiency in the long run without sacrificing the QoI levels attained. Specifically, we first introduce the novel concept of QoI-aware “sensor-to-task relevancy” to explicitly consider the sensing capabilities offered by an sensor to the IoT sensory environments, and QoI requirements required by a task. Second, we propose a novel concept of the “critical covering set” of any given task in selecting the sensors to service a task over time. Third, energy management decision is made dynamically at runtime, to reach the optimum for long-term application arrivals and departures under the constraint of their service delay. Finally, an extensive case study based on utilizing the sensing sensors to perform water quality monitoring is given to demonstrate the ideas and algorithms proposed in this paper, and a complete simulation is made to support all performance analysis.

In this paper, we propose an in-situ Crowd-aware Sensing Platform, called "CrowdMon", which envisions the cooperation among mobile users in highly crowded urban areas such as metro and square. CrowdMon establishes a spontaneous connection from co-located users in a semantic proximity and enables them to share contextual information such as location, ambient music, and mood of places. To the best of our knowledge, CrowdMon is the first attempt to support crowd-aware services at a platform level. We show interesting use cases of CrowdMon and an initial system design to realize the crowd-based context sharing.

No abstract available.

Participatory sensing is becoming more popular with the help of sensor-embedded smartphones to retrieve context-aware information for users. However, new challenges arise for the maintenance of the energy supply, the support of the quality-of-information (QoI) requirements, and the generation of maximum revenue for network operator, but with sparsely research exposure. This paper proposes a novel efficient network management framework to tackle the above challenges, where four key design elements are introduced. First is the QoI satisfaction index, where the QoI benefit the queries receive is quantified in relation to the level they require. Second is the credit satisfaction index, where the credits are used by the network operator to motivate the user participation, and this index is to quantify its degree of satisfaction. Third is the Gur Game-based distributed energy control, where the above two indexes are used as inputs to the mathematical framework of the Gur Game for distributed decision-making. Fourth is the dynamic pricing scheme, based on a constrained optimization problem to allocate credits to the participants while minimizing the necessary adaptation of the pricing scheme from the network operator. We finally evaluate the proposed scheme under an event occurrence detection scenario, where the proposed scheme successfully guarantees less than 7% detection outage, saves 80% of the energy reserve if compared with the lower bound solution, and achieves the suboptimum with only 4% gap if compared with optimal solution.

This paper considers the novel area of quality-of-information (QoI)-aware network management of multitasking wireless sensor networks (WSNs). Specifically, it provides an investigation of new task admission and resource utilization mechanisms for controlling the individual QoI provided to new and existing tasks using real-time feedback-based monitoring mechanisms. The paper describes three key design elements in support of the above: (a) the QoI satisfaction index of a task, which quantifies the degree to which the required QoI is satisfied by the WSN; (b) the QoI network capacity, which expresses the ability of the WSN to host a new task with specific QoI requirements without sacrificing the attained QoI levels of other existing tasks, and (c) an adaptive, negotiation-based admission control mechanism that reconfigures and optimizes the usage of network resources in order to optimally accommodate the QoI requirements of all tasks. Finally, extensive results are presented for assessing the performance of the proposed solution for the case of an intruder detection application scenario.

Controlled mobile helper nodes called data ferries have recently been proposed to bridge communications between disconnected nodes in a delay-tolerant manner. While existing work has explored various trajectory designs for the data ferry by assuming either static nodes or full observations at the data ferry, the problem remains open when the nodes are mobile and the ferry only has partial observations. In this paper, we investigate the problem of dynamic ferry mobility control under limited-range sensing. Assuming the data ferries are capable of sensing node presence within certain range and adjust their movements dynamically, we aim to design control policies that maximize the number of effective contacts. We provide a comprehensive model of the control framework using Partially Observable Markov Decision Process (POMDP), based on which we study the structure of the optimal policy and propose an efficient heuristic policy which shows significant improvement over the predetermined benchmark. To the best of our knowledge, this is the first data ferry control mechanism that can handle both stochastic node mobility and incomplete ferry observations.

Wireless Mesh Networks (WMNs) is a promising key technology for next generation wireless backhauling that is expected to support various types of applications with different quality-of-service (QoS) demands. Advanced antenna techniques, novel scheduling algorithms and routing schemes have attracted increased research interests aiming to optimize the performance of WMNs and satisfy their vast and diverse traffic requirements. However, in such networks, uncontrolled admission of arbitrary large number of data flows, together with the distributed and dynamic nature of WMNs, can highly increase the wireless channel interference with catastrophic subsequence for the whole network. To overcome this difficulty, we provide a mathematical model to analyze and estimate the real-time "route capacity" associated with time-varying QoS requirements. This allows us to take admission control decisions for new data flows in a way that we can increase the resource utilization while maintaining all good QoS levels for users with different grade-of-service (GoS). Numerical and extensive simulations results show that the proposed scheme achieves higher network goodput while it significantly reduces the outage and session blocking probability if compared to other techniques.

Wireless mesh networks (WMNs) is a promising key technology for next generation wireless backhauling that have recently attracted both the academic and industrial interest. Such networks are expected to have high throughput demands and support various types of applications with different quality-of-service (QoS) constraints. Opportunistic scheduling has been proven highly beneficial in such networks since it takes advantage of the dynamic nature of the channel between different wireless mesh routers. Recently a promising cross-layered framework has been proposed (IQoSR) that combines a distributed opportunistic scheduler with a multi-constrained QoS routing scheme and has been proven to outperform conventional layer 2 and 3 approaches. However, opportunistic scheduling cannot provide hard resource reservation; therefore, the admission of new flows in a route may jeopardize the QoS of the ongoing flows. In order to overcome this problem, in this work we introduce a connection admission control (CAC) scheme for different levels of QoS to efficiently manage the resources among existing and new flows. In this way, we improve the overall network performance and minimize the outage probability of the ongoing flows while we guarantee the required grade-of-service (GoS) to the underlying applications. Simulation analysis shows that the proposed scheme achieves lower blocking probability while it reduces the outage probability of the existing flows.

Cross-layer routing and scheduling algorithms design for wireless backhaul mesh network has attracted much research interest recently. The network is expected to support various types of applications with different quality of service (QoS) requirements from both routing and scheduling perspectives. Existing works do not efficiently integrate these QoS constraints in route discovery and maintenance phases and overlook the interaction between medium access control (MAC) and routing algorithms. In this work, we propose a novel cross- layer framework of QoS routing and distributed opportunistic scheduling for wireless mesh network, which provides resource reservation for QoS flows. Studies with different scheduling algorithms and routing protocols have shown that our algorithm successfully guarantees various QoS requirements and achieves higher network throughput when compared with other standard techniques.

Multihop wireless mesh networks (WMNs) are considered a promising technology to backhaul heterogeneous data traffic from wireless access networks to the wired Internet. WMNs are expected to support various types of applications with diverse quality of service (QoS) requirements, such as end-to-end packet delay, throughput, and packet-error-rate (PER). Recent works in this area are mainly concentrated on network layer routing algorithms with QoS provisioning that unfortunately cannot cooperate efficiently with existing medium access control (MAC) solutions to strictly guarantee multiple QoS constraints. This drawback may significantly deteriorate the and-to-end network performance and end-user experience especially for delay/throughput-sensitive applications such as voice-over IP (voIP) and interactive video. In this paper, we propose a fully distributed multi-constrained QoS scheduling algorithm to overcome this disadvantage. We show by simulation that the proposed scheduling scheme can efficiently organize the resources in physical (PHY) and MAC layers to successfully increase the network goodput, decrease the end-to-end (ETE) packet delay, and achieve less QoS outage probability if compared with other protocols.