Deep neural networks (DNNs) deployed on edge devices often suffer from severe performance degradation when exposed to dynamic and continually shifting environments. Test-time adaptation (TTA) has emerged as a promising solution by updating models online with incoming test data. However, edge deployment poses unique challenges: limited computational resources, latency caused by adaptation delays, and knowledge isolation across devices. The situation becomes even more complex in open-world scenarios, where the presence of unknown categories further disrupts adaptation. To overcome these limitations, we propose TeamTTA, a cloud-integrated framework designed for efficient multi-device collaboration open-set test-time adaptation. Specifically, TeamTTA aggregates reliable samples from multiple edge devices through crowdsourcing, uploads them to the cloud, and maintains a memory buffer for continual adaptation. A large vision model (LVM) in the cloud leverages its zero-shot generalization ability to filter out open-set samples and acts as a teacher model, distilling its knowledge into a replicated student edge model stored in the cloud. The adapted model parameters, or alternatively global statistics under poor network conditions, are then transmitted back to the edge devices for efficient inference. Extensive experiments on standard public TTA benchmarks, including corrupted and open-set datasets, show that TeamTTA achieves superior adaptation accuracy, robustness to distribution shifts, and communication efficiency, outperforming state-of-the-art TTA baselines. These results validate the effectiveness of integrating cloud-edge collaboration and LVM-driven knowledge distillation for real-world edge intelligence.

Continual adaptation to domain shifts at test time (CTTA) is crucial for enhancing the intelligence of deep learning enabled IoT applications. However, prevailing CTTA methods, which typically update all batch normalization (BN) layers, exhibit two memory inefficiencies. First, the reliance on BN layers for adaptation necessitates large batch sizes, leading to high memory usage. Second, updating all BN layers requires storing the activations of all BN layers for backpropagation, exacerbating the memory demand. Both factors lead to substantial memory costs, making existing solutions impractical for IoT devices. In this paper, we present FoCTTA, a low-memory CTTA strategy. The key is to automatically identify and adapt a few drift-sensitive representation layers, rather than blindly update all BN layers. The shift from BN to representation layers eliminates the need for large batch sizes. Also, by updating adaptation-critical layers only, FoCTTA avoids storing excessive activations. This focused adaptation approach ensures that FoCTTA is not only memory-efficient but also maintains effective adaptation. Evaluations show that FoCTTA improves the adaptation accuracy over the state-of-the-arts by 4.5%, 4.9%, and 14.8% on CIFAR10-C, CIFAR100-C, and ImageNet-C under the same memory constraints. Across various batch sizes, FoCTTA reduces the memory usage by 3-fold on average, while improving the accuracy by 8.1%, 3.6%, and 0.2%, respectively, on the three datasets.

For the fundamental linear programming (LP) problems, the simplex method remains popular, which usually requires an appropriate initial basis as a warm start to accelerate the solving process. Predicting an initial basis close to an optimal one can often accelerate the solver, but a closer initial basis does not always result in greater acceleration. To achieve better acceleration, we propose a GNN model based on a tripartite graph representation inspired by LP duality. This approach enables more effective feature extraction for general LP problems and enhances the expressiveness of GNNs. Additionally, we introduce novel loss functions targeting basic variable selection and basis feasibility, along with data preprocessing schemes, to further improve learning capability. In addition to achieving high prediction accuracy, we enhance the quality of the initial basis for practical use. Experimental results show that our approach greatly surpasses the state-of-the-art method in predicting initial basis with greater accuracy and in reducing the number of iterations and solving time of the LP solver.

Object detection, a fundamental task in computer vision, is crucial for various intelligent edge computing applications. However, object detection algorithms are usually heavy in computation, hindering their deployments on resource-constrained edge devices. Traditional edge-cloud collaboration schemes, like deep neural network (DNN) partitioning across edge and cloud, are unfit for object detection due to the significant communication costs incurred by the large size of intermediate results. To this end, we propose a Difficult-Case based Small-Big model (DCSB) framework. It employs a difficult-case discriminator on the edge device to control data transfer between the small model on the edge and the large model in the cloud. We also adopt regional sampling to further reduce the bandwidth consumption and create a discriminator zoo to accommodate the varying networking conditions. Additionally, we extend DCSB to video tasks by developing an adaptive sampling rate update algorithm, aiming to minimize computational demands without sacrificing detection accuracy. Extensive experiments show that DCSB can detect 97.26%-97.96% objects while saving 74.37%-82.23% network bandwidth, compared to cloud-only methods. Furthermore, DCSB significantly outperforms the latest DNN partitioning methods, reducing inference time by 92.60%-95.10% given an 8Mbps transmission bandwidth. In video tasks, DCSB matches the detection accuracy of leading video analysis methods while cutting the computational overhead by 40%.

Advancements in Earth observation sensors on low Earth orbit (LEO) satellites have significantly increased the volume of remote sensing images. This growth has led to challenges such as higher storage demands, downlink bandwidth stress, and transmission delays, particularly for real-time remote sensing image scene classification (RSISC). To address this, we propose a novel Satellite-Terrestrial Collaborative Scene Classification (STCSC) framework that integrates transmission and computation. The framework employs an attention-aware policy on the satellite, which adaptively determines the sequence of images and selection of image blocks for transmission, as well as these blocks' sampling rates. This policy is based on image complexity and the real-time data transmission rate, prioritizing blocks crucial for downstream tasks. On the ground, a classification model processes the received image blocks, balancing classification accuracy and transmission delay. Moreover, we have developed a comprehensive simulation system to validate the performance of our framework, including simulations of the satellite, transmission, and ground modules. Simulation results demonstrate that our STCSC framework can reduce transmission delay by 76.6% while enhancing classification accuracy on the ground by 0.6%. Additionally, our attention-aware policy is compatible with any ground classification model.

The Vision Transformer (ViT) excels in accuracy when handling high-resolution images, yet it confronts the challenge of significant spatial redundancy, leading to increased computational and memory requirements. To address this, we present the Localization and Focus Vision Transformer (LF-ViT). This model operates by strategically curtailing computational demands without impinging on performance. In the Localization phase, a reduced-resolution image is processed; if a definitive prediction remains elusive, our pioneering Neighborhood Global Class Attention (NGCA) mechanism is triggered, effectively identifying and spotlighting class-discriminative regions based on initial findings. Subsequently, in the Focus phase, this designated region is used from the original image to enhance recognition. Uniquely, LF-ViT employs consistent parameters across both phases, ensuring seamless end-to-end optimization. Our empirical tests affirm LF-ViT's prowess: it remarkably decreases Deit-S's FLOPs by 63% and concurrently amplifies throughput twofold. Code of this project is at https://github.com/edgeai1/LF-ViT.git.

Visual interpretation is extremely important in human geography as the primary technique for geographers to use photograph data in identifying, classifying, and quantifying geographic and topological objects or regions. However, it is also time-consuming and requires overwhelming manual effort from professional geographers. This paper describes our interdisciplinary team's efforts in integrating computer vision models with geographers' visual image interpretation process to reduce their workload in interpreting images. Focusing on the dune segmentation task, we proposed an approach featuring a deep dune segmentation model to identify dunes and label their ranges in an automated way. By developing a tool to connect our model with ArcGIS, one of the most popular workbenches for visual interpretation, geographers can further refine the automatically-generated dune segmentation on images without learning any CV or deep learning techniques. Our approach thus realized a non-invasive change to geographers' visual interpretation routines, reducing their manual efforts while incurring minimal interruptions to their work routines and tools they are familiar with. Deployment with a leading Chinese geography research institution demonstrated the potential of our approach in supporting geographers in researching and solving drylands desertification.

Segmenting dunes from remote sensing landforms images with deep vision models is promising by freeing geographers from manual visual interpretation tasks, making them more concentrated on the essential tasks in solving desertification challenges. However, geographers have reported that automated segmentation results may be not satisfactory though achieving high accuracy, implying there are potential gaps between pixel-level metrics and the utility in downstream geographic tasks. Therefore, pixel-wise metrics may be not proper in evaluating the deep dune segmentation in the geography domain, arising the necessity to develop domain-specific, human-centered measurements for deep dune segmentation. This paper first proposes a novel measurement based on geographers’ subjective judgments, which allows the evaluation of the alignment between deep dune segmentation models and geographical utility. We design an interactive framework integrating multiagent reinforcement learning (MARL) with geographers’ domain knowledge to improve models’ utility in the domain of geography. Our extensive experiments show that (1) our framework enables the interactive domain knowledge integration in the model-building process, and thus (2) the dune segmentation model better aligns with geographical utility, which ultimately improves the effectiveness of dune segmentation. We have deployed the framework with a number of geographers to support their various tasks including dune segmentation as a component. The results demonstrate our framework’s capabilities.

The rapid growth of deep learning-based services and applications underscores the need for efficient neural network model deployment. Traditional cloud-centric solutions, despite their computational power, face significant challenges such as high energy consumption, network transmission delays, and user privacy concerns. Conversely, performing high-performance inference on resource-constrained mobile devices, especially for tasks like advertising click prediction, presents its own set of difficulties. To address these challenges, we propose a device-cloud collaboration system utilizing a difficult-case discriminator. This system classifies input samples based on semantic information into difficult and simple cases. Difficult cases are processed in the cloud using a large model, while simple cases are handled on the device by a smaller model. This approach maximizes system resources and protects user privacy. Evaluations on public datasets show that our system significantly outperforms other advertisement methods in click prediction accuracy and uploading efficiency. Compared to the device-only approach, our system improves the area under the curve (AUC) by 8.9%, and compared to the cloud-centric approach, it reduces the upload ratio by 34%. Moreover, deploying our system on a specific smartphone demonstrates substantial improvements in private real datasets.

Object detection is crucial in video analytics pipelines, but there is a need to optimize deep neural networks (DNNs)-based object detection for resource-constrained Internet of Things (IoT) devices devices. The computational constraints inherent to the IoT device inevitably curtail its precision and real-time efficacy in the domain of object detection, with pronounced challenges arising, particularly when confronted with high-resolution video streams. To overcome these limitations, we propose UPD (Using Physical Dynamics), a novel on-device system that enables real-time and accurate object detection for high-resolution video streams. UPD employs a lightweight tracking algorithm for the detection of the majority of video frames, concurrently executing the object detector in a parallel fashion only in select instances. UPD addresses tracking errors by eliminating inaccurate feature points and correcting tracking results using physical information about the object. Unlike previous approaches that depend solely on the high-latency object detector to offset errors, our method is unaffected by the video resolution level. Extensive experiments demonstrate that UPD facilitates real-time analysis of high-resolution videos on IoT devices and significantly improves the overall accuracy (mIoU) compared to state-of-the-art DBT (Detection-Based-Tracking) frameworks, achieving a 100% accuracy improvement on three commonly used datasets. A video demo can be found at https://youtu.be/gKRQPHJ6gmY.

Deploying compute-intensive deep neural networks (DNNs) on resource-constrained end devices has become a prominent trend, enabling localized intelligence. However, efficiently deploying these DNNs at scale poses challenges. To address this, extensive research has focused on the early exit architecture based on convolutional neural networks (CNNs), which dynamically adapt network depth to reduce inference computation. Nevertheless, the sequential execution of all internal classifiers (ICs) and subsequent termination based on an exit criterion is inefficient. Motivated by these insights, we introduce a resolution-adaptive prediction network (RAPNet) architecture. RAPNet comprises a lightweight prediction network that captures global image features and an inference network integrated with an early exit architecture. The prediction network accurately determines the optimal IC position conditioned on the input images for efficient image classification. Additionally, we incorporate resolution-adaptive inference and feature fusion mechanisms by computational reuse, to effectively mitigate image spatial redundancy and improve the accuracy of ICs. We conduct extensive experiments across various data sets and architectures to demonstrate that RAPNet achieves a significantly better accuracy versus computational tradeoff than other recently proposed early exit methods. For instance, when using MobileNet as the base network, RAPNet achieves significant accuracy improvements of 12% and 5.7% on the Tiny Imagenet and CIFAR-100 data sets, respectively, surpassing other early exit methods with similar computational constraints.

The increasing deployment of multiple deep neural networks (DNNs) on edge devices is revolutionizing mobile vision applications, spanning autonomous vehicles, augmented reality, and video surveillance. These applications demand adaptation to contextual and environmental drifts, typically through fine-tuning on edge devices without cloud access, due to increasing data privacy concerns and the urgency for timely responses. However, fine-tuning multiple DNNs on edge devices faces significant challenges due to the substantial computational workload. In this paper, we present PatchLine, a novel framework tailored for efficient on-device training in the form of fine-tuning for multi-DNN vision applications. At the core of PatchLine is an innovative lightweight adapter design called patches coupled with a strategic patch updating approach across models. Specifically, PatchLine adopts drift-adaptive incremental patching, correlation-aware warm patching, and entropy-based sample selection, to holistically reduce the number of trainable parameters, training epochs, and training samples. Experiments on four datasets, three vision tasks, four backbones, and two platforms demonstrate that PatchLine reduces the total computational cost by an average of 55% without sacrificing accuracy compared to the state-of-the-art.

Technology mapping is a crucial step in the logic synthesis in chip design e.g. Field Programmable Gate Arrays (FPGAs) design, where a logic network is transformed into a K-bounded lookup tables (K-LUTs) network. Traditional mapping algorithms converges quickly to a suboptimal result, which limits the exploration capacity for further improvement. In this paper, we propose a new mapping method called Exploration-enhanced heuristics and Adaptive sequencing for Technology Mapping (EasyMap). EasyMap includes a pool of new heuristics and considers the mapping exploration as a conditional sequence optimization problem. During the mapping exploration procedure, heuristic algorithms with specific parameters are selected and applied sequentially. Our EasyMap outperforms the widely used IfMap in ABC by a significant margin. In particular, when optimizing area with a level constraint, EasyMap outperforms IfMap by reducing 9.1% more area on arithmetic circuits of the EPFL benchmark. Moreover, when optimizing area without level constraints at the same time, EasyMap can reduce 19% more area than IfMap on arithmetic circuits.

Recently, buoyed by advances in the space industry, low Earth orbit (LEO) satellites have become an important part of the Internet of Things (IoT). LEO satellites have entered the era of a big data link with IoT, how to deal with the data from the satellite IoT is a problem worthy of consideration. Conventional object detection method in optical remote sensing simply transmits the raw data to the ground. However, it ignores the properties of the images and the connection with the downstream task. To obtain efficient data transmission and accurate object detection, we propose a task-inspired satellite–terrestrial collaborative object detection framework called STCOD. It detects regions of interest (ROIs) and adopts a block-based adaptive sampling method to compress the background (BG) in optical remote sensing images by introducing satellite edge computing (SEC) on satellites. The STCOD framework also sets the transmission priority of image blocks according to their contributions to the task and uses fountain code to ensure the reliable transmission of important image blocks. We build a whole software simulation framework to validate our method, including the satellite module, the transmission module, and the terrestrial module. Extensive experimental results show that the STCOD framework can reduce the amount of downlink data decreased by 50.04% while losing the detection accuracy by 0.54%. In our simulated satellite–terrestrial link, the STCOD framework can reduce the number of satellite-to-terrestrial transmissions by half. When the packet loss rate is between 5% and 20%, the detection accuracy is lost only 0.05% to 0.5%.

Subjective reporting polarizes competing viewpoints. However, helping readers to recognize subjective content leads to more impartial discussions. Towards this end, we develop machine learning models that classify sentence objectivity. We contribute a set of linguistic rules for determining sentence objectivity collated from previous work. We also develop a labeled dataset with over 5000 sentences retrieved from various news sources. Further, we evaluate traditional machine learning classification models and artificial neural networks on our dataset. The best performing model, a convolutional neural network, achieved an accuracy of 85% and an AUC of 0.933. Using our subjective-objective sentence classification model, we implement Fact-or-Fiction, an end-to-end web system that highlights objective sentences in user text. Fact-or-Fiction provides additional information, such as links to related web pages and related previous submissions.