Flexible manufacturing is one of the core goals of Industry 4.0 and brings new challenges to current industrial control systems. Our detailed field study on auto glass industry revealed that existing production lines are laborious to reconfigure, difficult to upscale, and costly to upgrade during production switching. Such inflexibility arises from the tight coupling of devices, controllers, and control tasks. In this work, we propose a new architecture for industrial control systems named Control-as-a-Service (CaaS). CaaS transfers and distributes control tasks from dedicated controllers into network switches. By combining control and transmission functions in switches, CaaS virtualizes the whole industrial network to one Programmable Logic Controller (PLC). We propose a set of techniques that realize end-to-end determinism for in-network industrial control and a joint task and traffic scheduling algorithm. We evaluate the performance of CaaS on testbeds based on real-world networked control systems. The results show that the idea of CaaS is feasible and effective, and CaaS achieves absolute packet delivery, 42-45% lower latency, and three orders of magnitude lower jitter. We believe CaaS is a meaningful step towards the distribution, virtualization, and servitization of industrial control.

Time-Sensitive Networking (TSN) is foreseen as a foundational technology that enables Industry 4.0. It offers deterministic data transmission over Ethernet for critical applications such as industrial control and automotive systems. However, TSN is susceptible to hardware and software errors, necessitating an effective diagnostic system. Traditional network diagnostic tools are inadequate for TSN fault localization and classification due to the tightly coupled traffic and high precision requirements in TSN. In response, this paper presents TSNCard, a cross-cycle postcard-based diagnostic system tailored for Time-Aware Shaper (IEEE 802.1 Qbv) in TSN. TSNCard introduces a novel telemetry protocol that leverages the cyclical nature of TSN networks for data collection at each node. This protocol, coupled with dedicated analytic algorithms and hardware innovations within switches, forms a comprehensive system for TSN monitoring, fault localization and classification. Extensive experiments on both simulation and physical testbeds show that TSNCard can 100% detect fault location and type of the TSN misbehavior while adhering to industrial bandwidth restrictions. TSNCard not only bridges the gap in the TSN protocol stack, but also serves as a versatile toolkit for time-synchronized network analysis, paving the way for future research. The code is available athttps://github.com/MobiSense/TSNCard

Time-Sensitive Networking (TSN) is an enabling technology for Industry 4.0. Traffic scheduling plays a key role for TSN to ensure low-latency and deterministic transmission of critical traffic. As industrial network scales, TSN networks are expected to support a rising number of both time-triggered and event-triggered critical traffic (TCT and ECT). In this work, we present InNetScheduler, the first in-network TSN scheduling paradigm that boosts the throughput, i.e., number of scheduled data flows, of both traffic types. Different from existing approaches that conduct entire scheduling on the server, InNetScheduler leverages the computation resources on switches to promptly schedule latency-critical ECT, and delegate the computational-intensive TCT scheduling to server. The key innovation of InNetScheduler includes a Load-Aware Optimizer to mitigate ECT conflicts, a Relaxated ECT Scheduler to accelerate in-network computation, and End-to-End Determinism Guarantee to lower scheduling jitter. We fully implement a suite of InNetScheduler-compatible TSN switches with hardwaresoftware co-design. Extensive experiments are conducted on both simulation and physical testbeds, and the results demonstrate InNetScheduler’s superior performance. By unleashing the power of in-network computation, InNetScheduler points out a direction to extend the capacity of existing industrial networks.

Time-Sensitive Networking (TSN) is foreseen as a foundational technology that enables Industry 4.0. It offers deterministic data transmission over Ethernet for critical applications such as industrial control and automotive systems. However, TSN is susceptible to hardware and software errors, necessitating an effective diagnostic system. Traditional network diagnostic tools are inadequate for TSN fault localization due to the unique characteristics of TSN. In response, this paper presents TSNCard, a cross-cycle postcard-based diagnostic system tailored for TSN. TSNCard introduces a novel telemetry protocol that leverages the cyclical nature of TSN networks for data collection at each node. This protocol, coupled with dedicated analytic algorithms and hardware innovations within switches, forms a comprehensive system for TSN monitoring and fault localization. Extensive experiments on both simulation and physical testbeds show that TSNCard can 100% localize the root cause of the TSN misbehavior while adhering to industrial bandwidth restrictions. TSNCard not only bridges the gap in the TSN protocol stack, but also serves as a versatile toolkit for time-synchronized network analysis, paving the way for future research.

The edge-based multi-agent visual SLAM is crucial for emerging mobile applications like search-and-rescue, inventory automation, and industrial inspection. It uses a central node to manage the global map and schedule tasks for agents. However, as the number of agents increases, the system faces scalability challenges due to operational overhead, such as data redundancy, bandwidth consumption, and localization errors. In this paper, we introduce, a framework designed to enhance the scalability of collaborative visual SLAM service in edge offloading settings. consists of three system modules: a change log-based server-client synchronization mechanism, a priority-aware task scheduler, and a lean global map representation. These modules work together to address the challenges of data explosion problems. is open-source and compatible with the robotic operating system (ROS). Existing visual SLAM applications could incorporate through SwarmAPI, a set of well-packaged APIs, to compose SwarmMap’s function modules to enhance their performance and capacity in multi-agent scenarios. Comprehensive evaluations and a three-month case study at one of the world’s largest oilfields demonstrate that can serve 2$\times$more agents ($>$20 agents) than the state-of-the-arts with the same resource overhead, meanwhile maintaining an average trajectory error of 38$cm$, outperforming existing works by$>$55%.

Time-Sensitive Networking (TSN) has been considered the most promising network paradigm for time-critical applications (e.g., industrial control) and traffic scheduling is the core of TSN to ensure low latency and determinism. With the demand for flexible production increases, industrial network topologies and settings change frequently due to pipeline switches. As a result, there is a pressing need for a more efficient TSN scheduling algorithm. In this paper, we propose DeepScheduler, a fast and scalable flow-aware TSN scheduler based on deep reinforcement learning. In contrast to prior work that heavily relies on expert knowledge or problem-specific assumptions, DeepScheduler automatically learns effective scheduling policies from the complex dependency among data flows. We design a scalable neural network architecture that can process arbitrary network topologies with informative representations of the problem, and decompose the problem decision space for efficient model training. In addition, we develop a suite of TSN-compatible testbeds with hardware-software co-design and DeepScheduler integration. Extensive experiments on both simulation and physical testbeds show that DeepScheduler runs >150/5 times faster and improves the schedulability by 36%/39% compared to state-of-the-art heuristic/expert-based methods. With both efficiency and effectiveness, DeepScheduler makes scheduling no longer an obstacle towards flexible manufacturing.

Flexible manufacturing is one of the core goals of Industry 4.0 and brings new challenges to current industrial control systems. Our detailed field study on auto glass industry revealed that existing production lines are laborious to reconfigure, difficult to upscale, and costly to upgrade during production switching. Such inflexibility arises from the tight coupling of devices, controllers, and control tasks. In this work, we propose a new architecture for industrial control systems named Control-as-a-Service (CaaS). CaaS transfers and distributes control tasks from dedicated controllers into Time-Sensitive Networking (TSN) switches. By combining control and transmission functions in switches, CaaS virtualizes the industrial TSN network to one Programmable Logic Controller (PLC). We propose a set of techniques that realize end-to-end determinism for in-network industrial control and a joint task and traffic scheduling algorithm. We evaluate the performance of CaaS on testbeds based on real-world networked control systems. The results show that the idea of CaaS is feasible and effective, and CaaS achieves absolute packet delivery, 42-45% lower latency, and three orders of magnitude lower jitter. We believe CaaS is a meaningful step towards the distribution, virtualization, and servitization of industrial control.

Rapid advances in programmable network devices catalyzed the development of in-network computing, which is foreseen as a key enabler to empower the intelligence of production lines and mechanical arms in Industry 4.0. Various pioneering approaches have demonstrated the significant benefits of moving simple yet delay-sensitive industrial control tasks performed by servers to network switches. However, our detailed field study at a top-tier auto glass factory reveals that current practice fails to achieve a real-time and deterministic intelligent decision closure as leaving those complex yet essential planning tasks still on edge or cloud. In this paper, we design and implement a brand-new industrial switch, named Netopia, on a commercial Zynq platform through software and hardware co-design. Netopia enables planning and control to simultaneously perform on a network switch during communication. At the core of Netopia are three simple yet effective modules - a determinism guarantee mechanism, a computing acceleration scheme, and a packet deterministic forwarding framework that work hand-in-hand to ensure mechanical arms obtain intelligent control commands with low and deterministic latency. Comprehensive evaluations in industrial environments demonstrate that Netopia achieves an average end-to-end intelligent decision latency of 3.0ms with a jitter < 0.4ms, reduced by > 86% over existing works.

Real-time mobile video analysis like object detection and tracking is key to various household applications such as AR, cognitive assistance and smart home. Such applications rely on heavy DNN models, which are not suitable for mobile devices due to resource limitation. The long latency of cloud offloading is unacceptable for the real-time requirements, and the direct edge offloading relies on powerful edge servers, which is impractical for household scenarios. To solve this challenge, we take advantage of the computing devices that are low-cost or already exist in our lives, and propose Trine, a cloud-edge-device cooperated framework, in which complicated computation tasks are offloaded from the device to the cloud with the edge as the key bond to coordinate. In addition, due to the heterogeneity of edge devices, which leads to no one-fits-all algorithm that is optimal in all situations, we propose a profile-based algorithm to customize trackers for various edge devices. We implemented Trine on an android phone and three edge devices. The experiments demonstrate that Trine achieves 8-36% higher real-time accuracy and 25-89% higher robustness than state-of-the-art.

Time-Sensitive Networking (TSN) is the most promising network technology for Industry 4.0. A series of IEEE standards on TSN introduce deterministic transmission into standard Ethernet. Under the current paradigm, TSN can only schedule the deterministic transmission of time-triggered critical traffic (TCT), neglecting the other type of traffic in industrial cyber physical systems, i.e., event-triggered critical traffic (ECT). So in this work, we propose a new paradigm for TSN scheduling named E-TSN, which can provide deterministic transmission for both TCT and ECT. The three techniques of E-TSN, i.e., probabilistic stream, prioritized slot sharing, and prudent reservation, enable the deterministic transmission of ECT in TSN, and at the same time, protect TCT from the impacts of ECT. We also develop and make public a TSN evaluation toolkit to fill the gap in TSN study between algorithm design and experimental validation. The experiments show that E-TSN can reduce the latency and jitter of ECT by at least an order of magnitude compared to state-of-the-art methods. By enabling reliable and timely delivery of ECT in TSN for the first time, E-TSN can broaden the application scope of TSN in industry.

This paper studies the image fusion from multiple images taken by hand-held cameras with different exposures. The existing methods often generate unsatisfactory results, such as the blurring/ghosting artifacts due to the problematic handling of camera motions, dynamic contents, and inappropriate fusion of local regions (e.g., over or under exposed). They often require high quality image registration before fusion. However, the accurate alignment is hard to obtain in many scenarios, such as scenes with large depth variations and dynamic textures. Besides, high quality alignment is also time consuming. In this paper, we only enable a rough registration by a single homography and combine the inputs seamlessly to hide any possible misalignment. Specifically, we propose to use a Markov Random Filed (MRF) function for the labelling of all pixels, which assigns different labels to different aligned input images. During the labelling, we choose well-exposured regions and skip moving objects simultaneously. Then, we combine a Laplace image according to the labels and construct the fusion result by solving the Poisson equation. We present various challenging examples to demonstrate the effectiveness and practicability of our approach.