The 3D-printed mesh screen with dense penetrating pores is a new structure for massive manufacturing of molded pulp package products. However, some of the pores may be clogged by the printing material powder during the printing process. Such defects negatively affect the quality of the pulp packages produced using the mesh screen mold. To pinpoint the defects, we design a model-based robotic visual sensing system, called RoboCam, which uses a robotic arm to carry a high-resolution camera for full inspection of a mold consisting of joined mesh screens. To inspect the entire mold, RoboCam plans the camera poses to capture multiple images of the mold and render synthesized images as references for identifying the clogged pores. In particular, we propose novel designs to rectify the inherent run-time pose errors of the robotic system for ensuring the reference quality and to accelerate the reference rendering for reducing inspection latency. Extensive evaluation shows that RoboCam’s design outperforms various baselines, including three existing computer vision and convolution neural network-based inspection systems. RoboCam achieves a recall rate of 94.95% within 528 seconds latency for inspecting an entire mold with 13,000 designed pores.

Federated learning (FL) is a collaborative learning scheme to train deep learning models, where collaborating parties can consolidate their models without sharing local data with other parties, hence preserving data privacy. Nevertheless, when implementing FL in Industrial visual inspection (IVI), the constraints posed by limited data availability and the intricate nature of the inspection tasks significantly impact the performance of the resulting model. This paper introduces FedTR, a novel FL framework incorporating transfer learning designed for Autonomous IVI, focusing on the challenging task of identifying label defects through end-to-end text recognition. Transfer learning is a method that leverages the knowledge of a pre-trained model to adapt to a different dataset. FedTR initially trains the model using a publicly available dataset, after which performs the essential federated learning process with model fine-tuning on the distributed and limited private data. Extensive experiment results demonstrate the effectiveness and feasibility of FedTR on private ink cartridge datasets for label defect identification. FedTR achieves an end-to-end text recognition word-level accuracy of 95.5% and 94.2% on homogeneous and heterogeneous data respectively. Additionally, it attains performance levels that are on par with those achieved through centralized training.

Visual sensing has been widely adopted for quality inspection in production processes. This article presents the design and implementation of a smart collaborative camera system, called BubCam , for automated quality inspection of manufactured ink bags in Hewlett-Packard (HP) Inc.’s factories. Specifically, BubCam estimates the volume of air bubbles in an ink bag, which may affect the printing quality. The design of BubCam faces challenges due to the dynamic ambient light reflection, motion blur effect, and data labeling difficulty. As a starting point, we design a single-camera system that leverages various deep learning (DL)-based image segmentation and depth fusion techniques. New data labeling and training approaches are proposed to utilize prior knowledge of the production system for training the segmentation model with a small dataset. Then, we design a multi-camera system that additionally deploys multiple wireless cameras to achieve better accuracy due to multi-view sensing. To save power of the wireless cameras, we formulate a configuration adaptation problem and develop the single-agent and multi-agent deep reinforcement learning (DRL)-based solutions to adjust each wireless camera’s operation mode and frame rate in response to the changes of presence of air bubbles and light reflection. The multi-agent DRL approach aims to reduce the retraining costs during the production line reconfiguration process by only retraining the DRL agents for the newly added cameras and the existing cameras with changed positions. Extensive evaluation on a lab testbed and real factory trial shows that BubCam outperforms six baseline solutions including the current manual inspection and existing bubble detection and camera configuration adaptation approaches. In particular, BubCam achieves 1.3x accuracy improvement and 300x latency reduction compared with the manual inspection approach.

Various deep neural networks (DNNs) including convolutional neural networks (CNNs) and recurrent neural networks (RNNs) have shown appealing performance in various classification tasks. However, due to their large sizes, a single DNN often cannot fit into the memory of resource-constrained smart IoT sensors. This paper presents a DNN splitting framework calledNNFacetthat aims to run a DNN-based classification task on a total of$N$concurrent battery-based sensors observing the same physical process. We begin with determining the importance of all CNN filters or RNN units in learning each class. Then, an optimization problem divides the class set into$N$subsets and assigns them to the sensors, where the important CNN filters or RNN units associated with a class subset form a small model that is deployed to a sensor. Lastly, a multilayer perceptron is trained and deployed to a cloud or edge server, which yields the final classification result based on the low-dimensional features extracted by the sensors using their small models for the same observation. We apply NNFacet to three case studies of voice sensing, vibration sensing, and visual sensing. Extensive evaluation shows that NNFacet outperforms four baseline approaches in terms of system lifetime, latency, and classification accuracy.

Enabled by the increasingly available embedded hardware accelerators, the capability of executing advanced machine learning models at the edge of the Internet of Things (IoT) triggers interest of applying Artificial Intelligence of Things (AIoT) systems for industrial applications. The in situ inference and decision made based on the sensor data allow the industrial system to address a variety of heterogeneous, local-area non-trivial problems in the last hop of the IoT networks. Such a scheme avoids the wireless bandwidth bottleneck and unreliability issues, as well as the cumbersome cloud. However, the literature still lacks presentations of industrial AIoT system developments that provide insights into the challenges and offer lessons for the relevant research and industry communities. In light of this, we present the design, deployment, and evaluation of an industrial AIoT system for improving the quality control of HP Inc.’s ink cartridge manufacturing lines. While our development has obtained promising results, we also discuss the lessons learned from the whole course of the work, which could be useful to the development of other industrial AIoT systems for quality control in manufacturing.

Visual sensing has been increasingly employed in various industrial applications including manufacturing process monitoring and worker safety monitoring. This paper presents the design and implementation of a wireless camera system, namely, EFCam, which uses low-power wireless communications and edge-fog computing to achieve cordless and energy-efficient visual sensing. The camera performs image pre-processing and offloads the data to a resourceful fog node for advanced processing using deep models. EFCam admits dynamic configurations of several parameters that form a configuration space. It aims to adapt the configuration to maintain desired visual sensing performance of the deep model at the fog node with minimum energy consumption of the camera in image capture, pre-processing, and data communications, under dynamic variations of the monitored process, the application requirement, and wireless channel conditions. However, the adaptation is challenging due to the complex relationships among the involved factors. To address the complexity, we apply deep reinforcement learning to learn the optimal adaptation policy when a fog node supports one or more wireless cameras. Extensive evaluation based on trace-driven simulations and experiments show that EFCam complies with the accuracy and latency requirements with lower energy consumption for a real industrial product object tracking application, compared with five baseline approaches incorporating hysteresis-based and event-triggered adaptation.

Convolutional neural networks (CNNs) are increasingly adopted on resource-constrained sensors for in-situ data analytics in Internet of Things (IoT) applications. This paper presents a model split framework, namely, splitCNN, in order to run a large CNN on a collection of concurrent IoT sensors. Specifically, we adopt CNN filter pruning techniques to split the large CNN into multiple small-size models, each of which is only sensitive to a certain number of data classes. These class-specific models are deployed onto the resource-constrained concurrent sensors which collaboratively perform distributed CNN inference on their same/similar sensing data. The outputs of multiple models are then fused to yield the global inference result. We apply splitCNN to three case studies with different sensing modalities, which include the human voice, industrial vibration signal, and visual sensing data. Extensive evaluation shows the effectiveness of the proposed splitCNN. In particular, the splitCNN achieves significant reduction in the model size and inference time while maintaining similar accuracy, compared with the original CNN model for all three case studies.

Enabled by the increasingly available embedded hardware accelerators, the capability of executing advanced machine learning models at the edge of the Internet of Things (IoT) triggers wide interest of applying the resulting Artificial Intelligence of Things (AIoT) systems in industrial applications. The in situ inference and decision made based on the sensor data containing patterns with certain sophistication allow the industrial system to address a variety of heterogeneous, local-area non-trivial problems in the last hop of the IoT networks, avoiding the wireless bandwidth bottleneck and unreliability issues and also the cumbersome cloud. However, the literature still lacks presentations of industrial AIoT system developments that provide insights into the challenges and offer important lessons for the relevant research and engineering communities, no matter the development is successful or not. In light of this, we present the design, deployment, and evaluation of an industrial AIoT system for improving the quality control of Hewlett-Packard's ink cartridge manufacturing lines. While our development has obtained promising results, we also discuss the lessons learned from the whole course of the effort, which could be useful to the developments of other industrial AIoT systems.

Visual sensing has been increasingly employed in industrial processes. This paper presents the design and implementation of an industrial wireless camera system, namely, EFCam, which uses low-power wireless communications and edge-fog computing to achieve cordless and energy-efficient visual sensing. The camera performs image pre-processing (i.e., compression or feature extraction) and transmits the data to a resourceful fog node for advanced processing using deep models. EFCam admits dynamic configurations of several parameters that form a configuration space. It aims to adapt the configuration to maintain desired visual sensing performance of the deep model at the fog node with minimum energy consumption of the camera in image capture, pre-processing, and data communications, under dynamic variations of application requirement and wireless channel conditions. However, the adaptation is challenging due primarily to the complex relationships among the involved factors. To address the complexity, we apply deep reinforcement learning to learn the optimal adaptation policy. Extensive evaluation based on trace-driven simulations and experiments show that EFCam complies with the accuracy and latency requirements with lower energy consumption for a real industrial product object tracking application, compared with four baseline approaches incorporating hysteresis-based adaptation.