Municipal sewer asset management personnel without SQL expertise often rely on additional IT support to retrieve necessary information from their organization’s database, which hinders timely decision-making. This study presents a schema-guided RAG, a hybrid text-to-SQL framework that integrates structured knowledge graphs (KGs) for explicit join-path and relational reasoning with retrieval-augmented generation (RAG) for contextual grounding. Designed for domain-specific and multi-relational databases, the schema-guided RAG enables large language models (LLMs) to reason over complex relational structures while mitigating data scarcity through semantic retrieval. The framework was evaluated using both proprietary and open-source LLMs and applied in the sewer asset management domain. Results show consistent improvements in execution accuracy, logical form accuracy, and exact match, with particularly strong gains for queries requiring multi-table joins and nested logic. The schema-guided RAG offers an interpretable approach to natural language queries, supporting efficient, accurate, and explainable access to infrastructure data.

Due to the rapid expansion of printed circuit board (PCB) designs, accompanied by diverse design rules and specific constraints, there has been a substantial increase in manual design engineering efforts. To address this challenge, industries are seeking productivity improvements through automated placement techniques. However, existing placers primarily target VLSI placement and do not align well with PCBs’ unique characteristics. This mismatch arises from both the customization of PCBs and the complexity of the problem, which involves considering various constraints such as priorities, irregularities, and alignment. This paper introduces HiePlace, an efficient mathematical programming (MP)-based placement framework designed explicitly for PCBs. It aims to address the diverse constraints and achieve better performance. To address the issue of time-consuming computation in the direct MP-based algorithm, we present two innovative acceleration techniques: (1) In the initial stage, we introduce a dynamic programming approach to prioritize the placement of core components. This technique effectively reduces the solution space and enhances the overall placement quality. (2) Additionally, we propose a relaxation algorithm to minimize the number of boolean variables and further narrow down the solution space. This approach enables more efficient placement results by considering the problems specific constraints. Experimental results show that the proposed framework produces 7.7× speed up and 66% cost reduction.

Recent efforts of applying e-graphs in logic synthesis have shown promising results. Nevertheless, e-graph-based gate-level logic optimization suffers from inefficiency and limited extraction quality. In this article, we propose a fast parallel e-matching algorithm for speeding up e-graph rewriting, and an efficient netlist extraction framework with high quality of results in both area and delay. Experiments show that e-graph rewriting can be accelerated by up to 8.3× over a high-performance e-graph library, and our extraction framework achieves 11.0% and 1.0% improvements in size and level on average, compared to the best results of the state-of-the-art netlist extraction method.

As chip fabrication technology advances, the capacitive effects between wires have become increasingly pronounced, making crosstalk-induced incremental delay a serious issue. Traditional static timing analysis involves complex and iterative calculations through timing windows, requiring precise alignment of aggressor and victim nets, along with delay and slew estimations, which significantly increase runtime and licensing costs. In our work, we develop a Graph Neural Network framework to predict crosstalk-affected delays, focusing on the impacts of the coupling effect and overlapping nets. Moreover, we employ a curriculum learning strategy that gradually integrates aggressors with victims, improving model convergence through progressively complex scenarios. Experimental results show that our framework precisely predicts crosstalk-affected delays, matching commercial tools' performance with a fivefold speedup.

Efficient functional verification is crucial in the very-large-scale integration (VLSI) design flow. Existing processor-based emulation systems suffer from low efficiency due to the gap between partitioning and scheduling during compilation. To address the above concern, we propose ParSGCN, a scheduling-friendly emulation compilation flow that considers the objective of scheduling during partitioning. To incorporate the hard-to-perceive look-ahead information about scheduling, we embed it into a net cut probability distribution, which is easier to utilize. We estimate this probability distribution using a tailored variant of graph convolutional network (GCN) that is trained through a customized loss function and a large dataset of real-world compilation solutions. Additionally, we have developed a set of novel techniques to guide the emulation partitioning process using the estimated probability distribution. The proposed method is integrated into an industrial emulator and evaluated on large-scale designs with up to over 100 million cells. Comprehensive experimental results demonstrate the effectiveness of ParSGCN, showcasing an average improvement of 16.38%, 26.04%, and 19.52% in the best, worst, and median solution quality, respectively, based on 50 runs.

Composite Current Source (CCS) timing model plays an important role in modern static timing analysis (STA) because it precisely captures the timing behavior of a design at advanced nodes. However, CCS is extremely time-consuming due to its accurate but complicated timing models. To overcome this challenge, we introduce GCS-Timer, a GPU-accelerated CCS-based timing analysis algorithm. Unlike existing methods that perform model order reduction to trade accuracy for speed, GCS-Timer achieves high accuracy through a fast simulation-based analysis using GPU computing. Experimental results show that GCS-Timer can complete CCS analysis with better accuracy and achieve 3.2X faster runtime compared with a 16-threaded industrial standard timer. The source code is available at https://github.com/cuhk-eda/GCS-Timer.

Binary adders are a critical building block in integrated circuit (IC) design. In addition to the widely used 32/64/128-bit adders, large (1024/2048 bits) adders are important in applications such as cryptography. However, most current adder design methods target regular bitwidths, and cannot efficiently generate large adders with good performance. In practice, adders are often integrated into circuits such as a multiplier-accumulator (MAC), resulting in complex non-uniform input arrival times. To address these challenges, we propose a new algorithm for efficiently generating high-quality adders for non-uniform input arrival times. It is based on a novel divide-and-conquer-friendly problem formulation, and can effectively generate and maintain the most useful adder structures through dynamic programming. Experimental results show that it outperforms the current state-of-the-art methods in both quality and runtime. The adders generated by our algorithm have 2.8%, 8.3%, and 10.3% reductions in delay, area, and power, respectively, compared to those generated by a commercial synthesis tool.

Different people have different preferences when it comes to human-robot interaction. Therefore, it is desirable for the robot to adapt its actions to fit users’ preferences. Human feedback is essential to facilitating robot adaptation. However, when the task is complex or the robot action space is large, it requires a large amount of user feedback. ChatGPT is a powerful generative AI tool based on large language models (LLMs), which possesses a significant corpus of information obtained from human society, and exhibits robust proficiency in the comprehension and acquisition of natural language. Therefore, in this paper, we proposed a ChatGPT-powered adaptation system (ChatAdp) for human-robot interaction which requires less user feedback to achieve a good adaptation result. In the proposed ChatAdp, we use ChatGPT as a user simulator to provide feedback. We evaluated ChatAdp in a case study for context-aware conversation adaptation. The results are very promising. Our proposed method can achieve a mean success rate of 92% on the user’s natural language-described preferences after receiving 33 rounds of feedback from a user on average, which is only 2% of the number of states covered by the user preferences and outperforms the two baseline methods.

To ensure safe cooperative driving in mixed traffic with both manned and unmanned vehicles, it is crucial to understand and model the driving styles of human drivers. This paper explores how to develop accurate recognition of driving style and use that for the prediction of vehicle motion, which enables better performance in cooperative driving. A simulation testbed that consists of a driving simulator and a copilot is first introduced for the purpose of data collection and testing. A Long Short-Term Memory (LSTM)-based network that models human driving styles and predicts driving acceleration is developed. Standalone tests are conducted to examine the model performance in the simulation testbed. Finally, the model is evaluated in a series of merging experiments that involves 5 vehicles.

Existing conversational robots are mostly reactive in that the interactions are usually initiated by the users. With the knowledge of the environmental context such as people’s daily activities, robots can be more intelligent and proactive. In this paper, we proposed a context-aware conversation adaptation system (CACAS) for human-robot interaction (HRI). First, a context recognition module and a language processing module are developed to obtain the context information, user intent and slots, which become part of the state. Second, a reinforcement learning algorithm is developed to train an initial policy with a simulated user. User feedback data is collected through HRI using the initial policy. Third, a policy combining the reinforcement learning-based policy with the neural network-based policy is adapted based on the user feedback. We conducted both simulated user tests and real human subject tests to evaluate the proposed system. The results show that CACAS achieved a success rate of 85% in the real human subject test and 87.5% of participants were satisfied with the adaptation results. For the simulation test, CACAS had the highest success rate compared with the baseline methods.

3D Panoptic perception is essential for the understanding of real-world environment and plays an increasingly important role in the field of robotics. However, most existing methods heavily rely on image panoptic segmentation networks to acquire panoptic information of the environment, which is time-consuming and susceptible to interference. In this paper, we propose a novel and efficient panoptic mapping method based on multi-source information. Specifically, to improve the real-time performance of the system, we first apply lightweight object detection and semantic segmentation to extract 2D semantic and instance information from images. Second, a panoptic inference algorithm is designed that fully utilizes multi-source information, including geometry-based and learning-based information, to simultaneously reason about background and foreground objects in the environment. Finally, we take advantage of the scalability of the framework by introducing a multi-object tracking algorithm into the framework, thus providing the temporal information among consecutive frames to the data association module. Based on two popular datasets, extensive comparison experiments are conducted to illustrate the effectiveness of the proposed method. Experimental results show that compared with state-of-the-art panoptic mapping methods, the proposed method achieves superior performance in accuracy, real-timeness and stability. Furthermore, we also evaluate our method in real-world scenarios and CPU-only device to demonstrate the feasibility of its practical deployment.

Automatic pavement crack detection is an important task to ensure the functional performances of pavements during their service life. Inspired by deep learning (DL), the encoder-decoder framework is a powerful tool for crack detection. However, these models are usually open-loop (OL) systems that tend to treat thin cracks as the background. Meanwhile, these models can not automatically correct errors in the prediction, nor can it adapt to the changes of the environment to automatically extract and detect thin cracks. To tackle this problem, we embed closed-loop feedback (CLF) into the neural network so that the model could learn to correct errors on its own, based on generative adversarial networks (GAN). The resulting model is called CrackCLF and includes the front and back ends, i.e. segmentation and adversarial network. The front end with U-shape framework is employed to generate crack maps, and the back end with a multi-scale loss function is used to correct higher-order inconsistencies between labels and crack maps (generated by the front end) to address open-loop system issues. Empirical results show that the proposed CrackCLF outperforms others methods on three public datasets. Moreover, the proposed CLF can be defined as a plug and play module, which can be embedded into different neural network models to improve their performances.

Autonomous vehicles are gradually entering the transportation system. The traffic will become more heterogeneous since both autonomous and human-driven vehicles will share the roads. Cooperative driving, by promoting synchronized actions and shared situational awareness among vehicles, can significantly enhance driving safety. On the other hand, understanding human drivers is a pivotal step for cooperative driving in such mixed traffic environments, which facilitates effective interaction between human drivers and their vehicles. This paper presents a testbed that can be used to conduct research in intelligent and cooperative driving. The testbed consists of driving simulators, custom-designed copilots with an Artificial Intelligence engine, an optimization server, and a cloud database. The copilot is capable of sensing and understanding the human driver, the vehicle and the traffic. It can assist the driver by providing timely alerts on potential risks. Most importantly, it can communicate with other nearby vehicles for cooperative driving. Two case studies are presented to validate and evaluate the testbed. The first case study demonstrates the performance of the copilot in human distraction detection and driving assistance. The second case study focuses on cooperative driving between one human-driven vehicle and two connected autonomous vehicles in a lane-changing scenario. We expect this research testbed to be used in various research projects that involve human-driven vehicles and connected autonomous vehicles.

Rewriting is a widely used logic optimization approach incorporated in most commercial logic synthesis tools. In this paper, we present a new rewriting method based on And-Inverted Graph (AIG). Rather than focusing on cut rewriting, it considers a novel sub-structure called Maximum Fanout-Free Window (MFFW) and rewrites with a more compact implementation. Both exact synthesis and heuristic methods can be adopted to optimize MFFWs. A database dependent framework is proposed to store the optimal sub-structures to accelerate the processing. We further propose the semi-canonicalization to reduce the scale of the database, which could reduce more than 98% of the 4-input MFFW database. Extensive experiments on benchmark datasets demonstrate both the effectiveness and efficiency of our proposed framework.

Circuit partitioning is a critical step in hardware-assisted functional verification that involves splitting a circuit into multiple partitions and assigning them to specific hardware. However, partitioning a large circuit can require considerable computation resources and time, especially when complex hardware constraints are involved. Moreover, the path delay after partitioning can have a significant impact on verification efficiency, making early path delay prediction crucial for refining the circuit effectively. In this work, we propose a novel circuit partitioning predictor, named CPP, to rapidly and accurately predict the path delay after partitioning. To achieve this, we use circuit coarsening to develop a multi-level path representation and employ a convolutional neural network (CNN) that can capture both local and global path structures for delay prediction. Through extensive experiments on large industrial circuits, we demonstrate the superiority of our prediction framework.

Logic synthesis tools synthesize circuit structures to optimize specific targets given reasonable constraints and runtime using a set of well-defined operators. The efficiency of these operators is critical to achieving better runtime and optimization convergence. However, most synthesis operators are designed heuristically with fixed and sub-optimal traversal orders of nodes, cuts, and candidate subgraphs that are independent of circuit structures and functionalities. This leads to redundant computation and loss of optimization opportunities. Due to the spatial structure similarity, sub-circuits already synthesized in the same circuit contain meaningful information to guide more efficient synthesis for unvisited sub-circuits. Historical evaluation and gain features can learn from conflicts and be utilized to predict synthesis gain and prune invalid sub-circuits. Instead of using high-weight feature extraction and models, we utilize efficient prediction models with lightweight structural and functional features to reduce overhead. We thus propose a generalizable and dynamic scoring and pruning framework EffiSyn to accelerate logic synthesis operators while maintaining synthesis effectiveness. For example, we further improve the highly optimized operator drw by scoring and pruning invalid cuts and precomputed subgraphs. Extensive experiments on 20 public and industrial circuits validate that EffiSyn can accelerate drw by about 35% with negligible effectiveness loss or even with effectiveness improvement. Experiments over diverse circuits and synthesis sequences also validate the generalization of the proposed framework.

To achieve safe cooperative driving in mixed traffic of manned and unmanned vehicles, it is necessary to understand and model human drivers' driving behaviors. This paper proposed a Hidden Markov Model (HMM)-based method to analyze human driver's control and vehicle's dynamics; and then recognize the human driver's action, such as accelerating, braking, and changing lanes. With the knowledge of the human driver's actions, a probability model is used to predict the human-driven vehicle's acceleration. Such information on the driver behavior and the vehicle behavior can be used to achieve safer cooperative driving, which is realized using vehicle-to-vehicle (V2V) communication and model predictive control (MPC). The proposed method was tested and evaluated in our custom-built cooperative driving testbed. Experimental results show that the above driver action model is effective and accurate. A preliminary case study on a lane merging scenario is provided to further validate its effectiveness and capability.

Modeling a human-driven vehicle is a difficult subject since human drivers have a variety of stochastic behavioral components that influence their driving styles. We develop a cooperative driving framework to incorporate dif-ferent human behavior aspects, including the attentiveness of a driver and the tendency of the driver following advising commands. To demonstrate the framework, we consider the merging coordination between a human-driven vehicle and an autonomous vehicle (AV) in a connected environment. We propose a stochastic model predictive controller (sMPC) to address the stochasticity in human driving behavior and design coordinated merging actions to optimize the AV input and influence human driving behavior through advising commands. Simulation and human-in-the-loop (HITL) experimental results show that our formulation is capable of accommodating a distracted driver and optimizing AV inputs based on human driving behavior recognition.

This paper presents a cooperative driving testbed based on vehicle-to-vehicle (V2V) communication, which can be used for research in intelligent transportation systems, such as collision avoidance in mixed traffic of both human-driven vehicles and autonomous vehicles. To achieve the goal, an intelligent copilot is developed. The copilot can share the data regarding vehicle status, intention, etc, with other nearby vehicles through V2V communication. Several case studies are conducted to validate the proposed testbed and evaluate the performances of cooperative driving. When dangerous situations occur, the copilot solves the collision avoidance problem using Mixed Integer Programming (MIP), which either provides control commands to the autonomous vehicle, or advises the human driver to take action. Experimental results show that the safety and stability of the involved vehicles have been significantly enhanced. This cooperative driving testbed can be used by researchers to develop and test cooperative driving algorithms before they are deployed in real vehicles.

Robots need more intelligence to complete cognitive tasks in home environments. In this paper, we present a new cloud-assisted cognition adaptation mechanism for home service robots, which learns new knowledge from other robots. In this mechanism, a change detection approach is implemented in the robot to detect changes in the user’s home environment and trigger the adaptation procedure that adapts the robot’s local customized model to the environmental changes, while the adaptation is achieved by transferring knowledge from the global cloud model to the local model through model fusion. First, three different model fusion methods are proposed to carry out the adaptation procedure, and two key factors of the fusion methods are emphasized. Second, the most suitable model fusion method and its settings for the cloud-robot knowledge transfer are determined. Third, we carry out a case study of learning in a changing home environment, and the experimental results verify the efficiency and effectiveness of our solutions. The experimental results lead us to propose an empirical guideline of model fusion for the cloud-robot knowledge transfer.

Monitoring daily activities is essential for home service robots to take care of the older adults who live alone in their homes. In this article, we proposed a sound-based human activity monitoring (SoHAM) framework by recognizing sound events in a home environment. First, the method of context-aware sound event recognition (CoSER) is developed, which uses contextual information to disambiguate sound events. The locational context of sound events is estimated by fusing the data from the distributed passive infrared (PIR) sensors deployed in the home. A two-level dynamic Bayesian network (DBN) is used to model the intratemporal and intertemporal constraints between the context and the sound events. Second, dynamic sliding time window-based human action recognition (DTW-HaR) is developed to estimate active sound event segments with their labels and durations, then infer actions and their durations. Finally, a conditional random field (CRF) model is proposed to predict human activities based on the recognized action, location, and time. We conducted experiments in our robot-integrated smart home (RiSH) testbed to evaluate the proposed framework. The obtained results show the effectiveness and accuracy of CoSER, action recognition, and human activity monitoring.Note to Practitioners—This article is motivated by the goal to develop companion robots that can assist older adults living alone. Among many capabilities, monitoring human daily activities is an essential one for such robots. Though computer vision or wearable sensors-based methods have been developed by other researchers, they are not practical due to the privacy concern and intrusiveness. Sound-based daily activity recognition can address these concerns and offer a viable solution. In this regard, our proposed method adopts microphones on the robot and a small set of motion sensors distributed in the home. The proposed theoretical framework was tested in a small-scale mock-up apartment with promising results. Before such companion robots can be deployed to real homes for elderly care, there is a need to improve the robustness of the algorithms. More thorough tests in various realistic home environments should be conducted to fully evaluate the performance of the robots. In addition, privacy concern related to audio capture should be further mitigated.

We present a new framework for online dense 3D reconstruction of indoor scenes by using only depth sequences. This research is particularly useful in cases with a poor light condition or in a nearly featureless indoor environment. The lack of RGB information makes long-range camera pose estimation difficult in a large indoor environment. The key idea of our research is to take advantage of the geometric prior of Manhattan scenes in each stage of the reconstruction pipeline with the specific aim to reduce the cumulative registration error and overall odometry drift in a long sequence. This idea is further boosted by local Manhattan frame growing and the local-to-global strategy that leads to implicit loop closure handling for a large indoor scene. Our proposed pipeline, namely ManhattanFusion, starts with planar alignment and local pose optimization where the Manhattan constraints are imposed to create detailed local segments. These segments preserve intrinsic scene geometry by minimizing the odometry drift even under complex and long trajectories. The final model is generated by integrating all local segments into a global volumetric representation under the constraint of Manhattan frame-based registration across segments. Our algorithm outperforms others that use depth data only in terms of both the mean distance error and the absolute trajectory error, and it is also very competitive compared with RGB-D based reconstruction algorithms. Moreover, our algorithm outperforms the state-of-the-art in terms of the surface area coverage by 10-40 percent, largely due to the usefulness and effectiveness of the Manhattan assumption through the reconstruction pipeline.

Older adults who age in place face many health problems and need to be taken care of. Fall is a serious problem among elderly people. In this paper, we present the design and implementation of collaborative fall detection using a wearable device and a companion robot. First, we developed a wearable device by integrating a camera, an accelerometer and a microphone. Second, a companion robot communicates with the wearable device to conduct collaborative fall detection. The robot is also able to contact caregivers in case of emergency. The collaborative fall detection method consists of motion data based preliminary detection on the wearable device and video-based final detection on the companion robot. Both convolutional neural network (CNN) and long short-term memory (LSTM) are used for video-based fall detection. The experimental results show that the overall accuracy of video-based algorithm is 84%. We also investigated the relation between the accuracy and the number of image frames. Our method improves the accuracy of fall detection while maximizing the battery life of the wearable device. In addition, our method significantly increases the sensing range of the companion robot.

Social robots are coming to our homes and have already been used to help humans in a number of ways in geriatric care. This article aims to develop a framework that enables social robots to conduct regular clinical screening interviews in geriatric care, such as cognitive evaluation, falls’ risk evaluation, and pain rating. We develop a social robot with essential features to enable clinical screening interviews, including a conversational interface, face tracking, an interaction handler, attention management, robot skills, and cloud service management. Besides, a general clinical screening interview management (GCSIM) model is proposed and implemented. The GCSIM enables social robots to handle various types of clinical questions and answers, evaluate and score responses, engage interviewees during conversations, and generate reports on their well-being. These reports can be used to evaluate the progression of cognitive impairment, risk of falls, pain level, and so on by caregivers or physicians. Such a clinical screening capability allows for early detection and treatment planning in geriatric care. The framework was developed and implemented on our 3-D-printed social robot. It was tested on 30 older adults with different ages, achieved satisfying results, and received their high confidence and trust in the use of this robot for human well-being assessment.Note to Practitioners—This article is motivated by the goal of using a social robot to perform geriatric well-being assessment through clinical screening interviews. In order to conduct clinical screening interviews, the social robot needs the following essential features: having a verbal conversational interface, adapting to different types of clinical screening interviews, scoring and evaluating answers, having nondirective listening responses, and enabling directive listening responses. The proposed general clinical screening interview management (GCSIM) model demonstrates these capabilities on the social robot. The robot can give structured clinical screening interviews with different question–answer sheets. This will help advance assistive technologies for use by geriatric physicians, nurses, and social service professionals to keep older adults healthy, safe, and independent at home. Robots will become more and more essential in working alongside geriatric practitioners to help monitor older adults at home and to provide early detection and warning of cognitive/mental health problems, falls’ risk, and so on. This early detection property can improve quality-of-care and help older adults remain living at home.

Distracted driving is one of the main contributors to traffic accidents. This paper proposes a deep learning approach to detecting multiple distracted driving behaviors. In order to obtain more accurate detection results, a synchronized image recognition system based on two cameras is designed, by which the body movements and face of the driver are monitored respectively. The images captured from driver's body and face areas are fed to two Convolutional Neural Networks (CNNs) simultaneously to ensure the performance of classification. The data collection and validation processes of the proposed distraction detection approach were conducted on a laboratory-based assisted driving testbed to provide near-realistic driving experiences. Our dataset includes distracted and safe driving images of the drivers. Furthermore, we developed a meaningful and practical application of a voice-alert system that alerts the distracted driver to focus on the driving task. We evaluated VGG-16, ResNet, and MobileNet-v2 networks for the proposed approach. Experimental results show that by using two cameras and VGG-16 networks, we can achieve a recognition accuracy of 96.7% with a computation speed of 8 fps.

There are two common paradigms for indoor scene understanding, pixel-level labeling and bounding box generation. The two tasks have a complementary nature but are normally achieved separately with different computational flows. We propose a novel method to bridge the two tasks by creating category-specific 3D bounding box hypotheses from score-maps of any deep networks trained on pixel-level semantic labels along with depth data. Those hypotheses can be further used to locate all objects as different non-overlapping bounding boxes by incorporating high-level knowledge, such as common room settings, co-existence or co-exclusiveness etc. We develop an objective function that involves confidence scores and the depth visibility to initialize and optimize multiple hypotheses for each category-specific score map. Experiment results show that our method significantly outperforms direct bounding box generation using pixel-level labeling.

We present an online framework for dense 3D reconstruction of indoor scenes using sequential Manhattan keyframes. We take advantage of the global geometry of indoor scenes extracted by Manhattan frames and pose optimization to enhance the accuracy and robustness of the reconstructed models. During sequential reconstruction, a Manhattan frame is extracted for each keyframe after surface normal adjustment, and used for a Manhattan keyframe-based planar alignment to initialize the surface registration while a pose graph optimization is used to refine camera poses. The final model is created by integrating the Manhattan keyframes into the unified volumetric model using refined pose estimations. Experimental results demonstrate the advantage of our geometry-based approach to reduce the cumulative registration error and overall geometric drift.

Smart homes can provide complementary information to assist home service robots. We present a robotic misplaced item finding (MIF) system, which uses human historical trajectory data obtained in a smart home environment. First, a multi-sensor fusion method is developed to localize and track a resident. Second, a path-planning method is developed to generate the robot movement plan, which considers the knowledge of the human historical trajectory. Third, a real-time object detector based on a convolutional neural network is applied to detect the misplaced item. We present MIF experiments in a smart home testbed and the experimental results verify the accuracy and efficiency of our solution.

The goal of this paper is to localize a resident in indoor environments by using motion information from distributed environmental sensors and body activity information from wearable sensors. The passive infrared sensor nodes distributed in a home provide binary information about human motion in their field of views, while the wearable inertial measurement unit sensor node collects motion data that can be used in body activity recognition, walking velocity, and heading estimation. Basic human activities such as sitting, sleeping, standing, and walking are recognized. We proposed a particle filter-based sensor fusion algorithm that takes advantage of the human location/activity correlation in indoor environments to increase the localization accuracy. Experiments were conducted in a mock apartment testbed. We used the ground truth data obtained from a motion capture system to evaluate the results.

With a goal to improve transportation safety, this paper proposes a collaborative driving framework based on assessments of both internal and external risks involved in vehicle driving. The internal risk analysis includes driver drowsiness detection and driver intention recognition that helps to understand the human driver's behavior. Steering wheel data and facial expression are used to detect the driver's drowsiness. Hidden Markov models are adapted to recognize the driver's intention using the vehicle's lane position, control, and state data. For the external risk analysis, a co-pilot utilizes a collision avoidance system to estimate the collision probability between the ego vehicle and other nearby vehicles. Based on the risk analyses, we design a novel collaborative driving scheme by fusing the control inputs from the human driver and the co-pilot to obtain the final control input for the ego vehicle under different circumstances. The proposed collaborative driving framework is validated in an assisted-driving testbed, which enables both autonomous and manual driving capabilities.

This paper proposes a driver assistance system with a collaborative control framework between the human driver and a Collision Avoidance System (CAS) for vehicles on the road while considering the driver drowsiness status. Driver drowsiness detection is performed with two inputs: driver's facial data from a camera mounted in front of the driver and steering wheel data from the car controller system. We use the driver's drowsiness state as an input to the collaborative control framework in which the CAS algorithm runs in parallel with the human control and only intervenes under certain situations to assist the human driver. Experiments were performed on a simulated vehicle driving testbed to evaluate our drowsiness detection system and demonstrate the effectiveness of the proposed collaborative control framework.

We propose a new cuboid matching algorithm for robust object detection in RGB-D images from an indoor scene. Unlike traditional bounding boxes, a cuboid is more flexible and accurate to represent the object's orientation and basic geometry that can serve as an informative mid-level representation for scene understanding. However, over-detection and miss-detection are common problems when the scene is too cluttered and has many irrelevant planar surfaces. We approach these problems from two perspectives. First, we apply a few planar features to improve initial plane generation and to select dominant plane candidates for cuboid initialization. Second, cuboid candidates are optimized to their local fitness involving both color and depth features. The experimental results show significant improvements over the state-of-art method both quantitatively and qualitatively.

We propose an efficient 3D modeling method to support real-time volumetric reconstruction of indoor scenes based on sequential depth sequences captured from a RGB-D camera. Specifically, we want to reduce the cumulative error from sequential ICP registration due to noise and outliers in the depth data. We take advantage of the Manhattan frame assumption valid in most indoor scenes that can be used to facilitate large scale 3D surface registration. In our approach, the Manhattan frame is extracted from each depth frame and used for plane-to-plane frame alignment to initialize point-to-plane ICP surface registration. Experimental results on three different indoor datasets including LIDAR ground-truth data demonstrate the advantages of the proposed algorithm over the original ICP-based approaches to volumetric reconstruction.

Map building is a fundamental task in many robotic applications. In this paper, we propose a novel approach for 3D mapping of indoor environments which allows a robot avatar to collaborate with a human seamlessly through a virtual reality (VR) device. The 3D map is created using the 3D data from an RGB-D camera mounted on the robot and simultaneously transmitted to a remote server, and then rendered to the VR device. On the other hand, the intentions of the user are inferred using the motion of the head movement based on hidden Markov models (HMMs), and then interpreted into commands to control the robot. We implement the proposed approach based on a modified Pioneer robot platform. The experimental results show the feasibility of the proposed system.

Water is a highly abundant nutrient in the human body and monitoring of its regulation is essential to keep the body hydrated. A number of critical health conditions including swelling of the brain and short/long term memory loss are associated with poor or excessive drinking habits. This can be prevented with the use of a real time hydration monitoring system. In this paper we presented AutoHydrate, a wearable hydration monitoring system which continuously monitors the drinking activities and daily fluid requirements of the user through automatic detection of drinking and body activities. The system is built using a throat microphone for collecting acoustic signals, a smartwatch for collecting body activity, an embedded computer for processing the signals and sending recommendation to a smartphone app in real time for an interactive information display. After different time, frequency and cepstral domain features are extracted from the signals, drinking activities are classified using Support Vector Machine (SVM) and body activity is classified using Gradient Boosting Decision Tree algorithm. The Dietary Reference Intake standard is followed for recommending the amount of fluid required using our detection. Based on our experimental results on 8 subjects, a Drinking detection accuracy of 91.5% and Body activity classification accuracy of 89.12% are obtained. Results show that our system is feasible for real time monitoring of body hydration.

In this correspondence paper, we develop a teleoperated robotic 3-D mapping (TeRoM) system which enables efficient human-guided mapping of remote environments for realistic rendering and visualization. First, the hardware design of the TeRoM system is proposed which is based on a Pioneer mobile robot platform and a rotating RGB-D camera. A client/server architecture is developed to allow the data to be processed in a remote server, which makes it possible to implement 3-D mapping on robots with limited resources. Second, a 3-D map is created in real-time while an operator controls the robot and the pan-tilt unit remotely using a joystick. Then the map is converted into a mesh using the marching cubes algorithm and optimized to reduce the data volume. Finally the mesh is imported and rendered in a 3-D rendering engine for interactive and intuitive display. We evaluate the performance of the TeRoM system in terms of accuracy, processing speed, reliability, and manipulability.

This paper proposes and implements an auditory perception platform for a home service robot that serves the elderly living alone at home. A layered architecture for the proposed platform was developed to realize the various auditory perception capabilities while enabling a remote caregiver to involve in the sound event recognition process. We successfully implemented the software for this architecture that realizes robot services and auditory services for developing high level auditory applications. The robot is able to estimate the sound source position and recognize human speech in the room with multiple sound sources, as well as to collaborate with the caregiver on sound event recognition. Our experimental results validated the proposed platform.

Teaching robots manipulative skills through human demonstration is an important research problem and can be used to quickly program robots in future manufacturing industries. To understand human demonstration, manipulative actions need to be recognized. To improve the recognition performance, we use three kinds of sensors to capture the motion and force involved in the fine manipulative actions. In addition, by taking advantage of the action/object correlation, the recognition accuracy can be further improved. In the proposed approach, important features for individual actions are selected first. Hidden Markov Models (HMMs) are employed to characterize the temporal changes. Then, a Bayesian model is adopted to model the object/action dependency. Our approach was evaluated through experiments on assembly tasks. The experimental results show that the proposed approach can recognize manipulative actions effectively.

The papers in this special section present the most recent research work that showcases the state-of-the-art of human-centered computing and its potential applications in developing truly smart home automation systems.

Detecting behavioral anomalies in human daily life is important to developing smart assisted-living systems for elderly care. Based on data collected from wearable motion sensors and the associated locational context, this paper presents a coherent anomaly detection framework to effectively detect different behavioral anomalies in human daily life. Four types of anomalies, including spatial anomaly, timing anomaly, duration anomaly, and sequence anomaly, are detected using a probabilistic theoretical framework. This framework is based on complex activity recognition using dynamic Bayesian network modeling. The maximum-likelihood estimation algorithm and Laplace smoothing are used in learning the parameters in the anomaly detection model. We conducted experimental evaluation in a mock apartment environment, and the results verified the effectiveness of the proposed framework. We expect that this behavioral anomaly detection system can be integrated into future smart homes for elderly care.

This paper presents an integrated learning framework that enables humanoid robots to perform human-robot collaborative manipulation tasks. Specifically, a table-lifting task performed jointly by a human and a humanoid robot is chosen for validation purpose. The proposed framework is split into two phases: 1) phase I-learning to grasp the table and 2) phase II-learning to perform the manipulation task. An imitation learning approach is proposed for phase I. In phase II, the behavior of the robot is controlled by a combination of two types of controllers: 1) reactive and 2) proactive. The reactive controller lets the robot take a reactive control action to make the table horizontal. The proactive controller lets the robot take proactive actions based on human motion prediction. A measure of confidence of the prediction is also generated by the motion predictor. This confidence measure determines the leader/follower behavior of the robot. Hence, the robot can autonomously switch between the behaviors during the task. Finally, the performance of the human-robot team carrying out the collaborative manipulation task is experimentally evaluated on a platform consisting of a Nao humanoid robot and a Vicon motion capture system. Results show that the proposed framework can enable the robot to carry out the collaborative manipulation task successfully.

Scalar field mapping has many applications including environmental monitoring, search and rescue, etc. In such applications, there is a need to achieve a certain level of confidence regarding the estimates of the scalar field. In this paper, a cooperative and active sensing framework is developed to enable scalar field mapping using multiple mobile sensor nodes. The cooperative and active controller is designed via the real-time feedback of the sensing performance to steer the mobile sensors to new locations in order to improve the sensing quality. During the movement of the mobile sensors, the measurements from each sensor node and its neighbors are fused with the corresponding confidences using distributed consensus filters. As a result, an online map of the scalar field is built while achieving a certain level of confidence of the estimates. We conducted computer simulations to validate and evaluate our proposed algorithms.

A growing concern with deploying Unmanned Aerial Vehicles (UAVs) in urban environments is the potential violation of human privacy, and the backlash this could entail. Therefore, there is a need for UAV path planning algorithms that minimize the likelihood of invading human privacy. We formulate the problem of human-aware path planning as a nonstationary Markov Decision Process, and provide a novel model-based reinforcement learning solution that leverages Gaussian process clustering. Our algorithm is flexible enough to accommodate changes in human population densities by employing Bayesian nonparametrics, and is real-time computable. The approach is validated experimentally on a large-scale long duration experiment with both simulated and real UAVs.

This paper proposes and implements a framework for human-robot collaboration in a Mobile Visual Sensor Network (MVSN). A collaborative architecture for the proposed human-integrated MVSN was developed to allow the human operator and robots to collaborate to perform surveillance tasks. We successfully implemented the MVSN so the user can control the deployment of the mobile sensors through his head movement. We also explored using computer vision techniques and navigation techniques on the robot nodes to conduct active human target detection. The robot nodes, therefore, are able to detect human faces while exploring the unknown environment, and then relay the face images to the operator for target recognition. In this way, humans and robots can complement each other to accomplish surveillance tasks. Our experimental results validated the proposed framework.

Acquiring robot assembly skills through human demonstration is a challenging problem. To achieve this goal, not only the actions and objects have to be shown to the robot, but also the effect of the action needs to be estimated. Recognizing the subtle assembly actions is a non-trivial task, and it is difficult to estimate the effect of the action on the assembly parts due to the small part sizes. In this paper, with a RGB-D camera, we build a Portable Assembly Demonstration (PAD) system which can recognize the part/tool used, the action applied and the assembly state characterizing the spatial relationship between the parts. The experiment results proved that this PAD system can generate an assembly script with good accuracy in object and action recognition as well as assembly state estimation.

Drowsiness is one of the main causes of severe traffic accidents occurring in our daily life. In order to reduce the number of drowsiness-induced accidents, various researches have been conducted with the aim of finding practical and non-invasive drowsiness detection systems by using behavioral measuring techniques. Many of the previous works on behavioral measuring techniques have mainly focused on the analysis of eye closure and blinking of the driver. It is recently that more attention started to shift to inclusion of other facial expressions and only few, among those researches, have been done on the analysis of temporal dynamics of facial expressions for drowsiness detection. In this paper we propose a new method of analyzing the facial expression of the driver through Hidden Markov Model (HMM) based dynamic modeling to detect drowsiness. We have implemented the algorithm using a simulated driving setup. Experimental results verified the effectiveness of the proposed method.

One of the important tasks for bridge maintenance is bridge deck crack inspection. Traditionally, a human inspector detects cracks using his/her eyes and marks the location of cracks manually. However, the accuracy of the inspection result is low due to the subjective nature of human judgement. We propose a crack inspection system that uses a camera-equipped mobile robot to collect images on the bridge deck. In this method, the Laplacian of Gaussian (LoG) algorithm is used to detect cracks and a global crack map is obtained through camera calibration and robot localization. To ensure that the robot collects all the images on the bridge deck, a path planning algorithm based on the genetic algorithm is developed. The path planning algorithm finds a solution which minimizes the number of turns and the traveling distance. We validate our proposed system through both simulations and experiments.

This brief proposes an output tracking control for a class of discrete-time nonlinear systems with disturbances. A standard neural network model is used to represent discrete-time nonlinear systems whose nonlinearity satisfies the sector conditions. H∞ control performance for the closed-loop system including the standard neural network model, the reference model, and state feedback controller is analyzed using Lyapunov-Krasovskii stability theorem and linear matrix inequality (LMI) approach. The H∞ controller, of which the parameters are obtained by solving LMIs, guarantees that the output of the closed-loop system closely tracks the output of a given reference model well, and reduces the influence of disturbances on the tracking error. Three numerical examples are provided to show the effectiveness of the proposed H∞ output tracking design approach.

In this paper, we propose and develop a framework for automatic switching of manual driving and autonomous driving based on driver drowsiness detection. We first present the scale-down intelligent transportation system (ITS) testbed. This testbed has four main parts: an arena; an indoor localization system; automated radio controlled (RC) cars; and roadside monitoring facilities. Second, we present the drowsiness detection algorithm which integrates facial expression and racing wheel motion to recognize driver drowsiness. Third, a manual and autonomous driving switching mechanism is developed, which is triggered by the detection of drowsiness. Finally, experiments were performed on the ITS testbed to demonstrate the effectiveness of the proposed framework.

In this paper, autonomous mobile sensor networks are deployed to measure a scalar field and build its map. We develop a novel method for multiple mobile sensor nodes to build this map using noisy sensor measurements. Our method consists of two parts. First, we develop a distributed sensor fusion algorithm by integrating two different distributed consensus filters to achieve cooperative sensing among sensor nodes. This fusion algorithm has two phases. In the first phase, the weighted average consensus filter is developed, which allows each sensor node to find an estimate of the value of the scalar field at each time step. In the second phase, the average consensus filter is used to allow each sensor node to find a confidence of the estimate at each time step. The final estimate of the value of the scalar field is iteratively updated during the movement of the mobile sensors via weighted average. Second, we develop the distributed flocking-control algorithm to drive the mobile sensors to form a network and track the virtual leader moving along the field when only a small subset of the mobile sensors know the information of the leader. Experimental results are provided to demonstrate our proposed algorithms.

In this paper, H∞ synchronization and state estimation problems are considered for different types of chaotic systems. A unified model consisting of a linear dynamic system and a bounded static nonlinear operator is employed to describe these chaotic systems, such as Hopfield neural networks, cellular neural networks, Chua's circuits, unified chaotic systems, Qi systems, chaotic recurrent multilayer perceptrons, etc. Based on the H∞ performance analysis of this unified model using the linear matrix inequality approach, novel state feedback controllers are established not only to guarantee exponentially stable synchronization between two unified models with different initial conditions but also to reduce the effect of external disturbance on the synchronization error to a minimal H∞ norm constraint. The state estimation problem is then studied for the same unified model, where the purpose is to design a state estimator to estimate its states through available output measurements so that the exponential stability of the estimation error dynamic systems is guaranteed and the influence of noise on the estimation error is limited to the lowest level. The parameters of these controllers and filters are obtained by solving the eigenvalue problem. Most chaotic systems can be transformed into this unified model, and H∞ synchronization controllers and state estimators for these systems are designed in a unified way. Three numerical examples are provided to show the usefulness of the proposed H∞ synchronization and state estimation conditions.

Semantic information can help both humans and robots to understand their environments better. In order to obtain semantic information efficiently and link it to a metric map, we present a semantic mapping approach through human activity recognition in an indoor human-robot coexisting environment. An intelligent mobile robot platform can create a 2D metric map, while human activity can be recognized using motion data from wearable motion sensors mounted on a human subject. Combined with pre-learned models of activity-to-furniture type association and robot pose estimates, the robot can determine the distribution of the furniture types on the 2D metric map. Simulations and real world experiments demonstrate that the proposed method is able to create a reliable metric map with accurate semantic information.

In this paper, we propose and develop a small-scale research platform for intelligent transportation systems (ITSs). Our platform has four main parts, i.e., an arena, an indoor localization system, automated radio-controlled (RC) cars, and roadside monitoring facilities. First, to mimic traffic environments, we build an arena with a wooden floor, mock buildings, and streets. Second, to facilitate feedback control for trajectory following, an indoor localization system is set up to track the RC cars. Third, both autonomous driving RC cars and human driving RC cars are developed, based on an automated RC car design. The automated RC cars can receive control signals from a computer through an Xbee RF module and control the front and rear wheels through motors. A new control algorithm is developed to allow the RC cars to track predefined trajectories. Finally, we implement an example of roadside monitoring, which uses a fish-eye camera associated with advanced video processing for image segmentation, object identification, and tracking. Experiments are performed to demonstrate the effectiveness of the designed platform. We also discuss possible ITS research problems that can be studied in this testbed.

Surface manufacturing is a process of adding material to or removing material from the surfaces of a part, where industrial robots are usually used. Robot programming for these applications is still time consuming and costly. Typical teaching methods are not cost effective and efficient. There are many offline programming methods developed to reduce the robot programming effort. However, these methods suffer many practical issues, such as cable/hose tangling, robot configuration, collision and reachability. To solve these problems, this paper discusses a new method to generate robot programs based on existing ones in the database, which contains not only the robot paths, but also the programmers' knowledge and process parameters. Since industrial robots have been used in production for decades, there are many robot programs for different parts generated by the robot programmers. Some of these parts are quite similar, such as car bodies. Therefore, these robot programs can be transformed to generate new ones with some adjustments based on the geometry of new parts. In this paper, a robot path generation method is developed based on the existing robot paths in the database. Experiments were performed to validate the developed methodology. The results are very promising in reducing the programming efforts in surface manufacturing.

One of the important tasks for bridge maintenance is bridge deck crack inspection. Traditionally, a human inspector detects cracks using his/her eyes and finds the location of cracks manually. Thus the accuracy of the inspection result is low due to the subjective nature of human judgement. We propose a system that uses a mobile robot to conduct the inspection, where the robot collects bridge deck images with a high resolution camera. In this method, the Laplacian of Gaussian algorithm is used to detect cracks and the global crack map is obtained through camera calibration and robot localization. To ensure that the robot collects all the images on the bridge deck, we develop a complete coverage path planning algorithm for the mobile robot. We compare it with other path planning strategies. Finally, we validate our proposed system through experiments and simulation.

Traditionally the leader or follower role of the robot in a human-robot collaborative task has to be predetermined. However, humans performing collaborative tasks can switch between or share the leader-follower roles effortlessly even in the absence of audio-visual cues. This is because humans are capable of developing a mutual understanding while performing the collaborative task. This paper proposes a framework to endow robots with a similar capability. Behavior of the robot is controlled by two types of controllers such as reactive and proactive controllers each giving the robot follower and leader characteristics respectively. Proactive actions are based on human motion prediction. We propose that the role of the robot can be governed by the confidence of prediction. Hence, the robot can determine its role during the task autonomously and dynamically. The framework is demonstrated and evaluated through a table-lifting task. Experimental results confirm that the proposed system improves the overall task performance.

In this paper, we proposed a method to recognize complex human daily activities including body activities and hand gestures simultaneously in an indoor environment. Three wearable motion sensors are attached to the right thigh, the waist, and the right hand of a person, while an optical motion capture system is used to obtain his/her location information. A three-level dynamic Bayesian network is implemented to model the intra-temporal and inter-temporal constraints among the location, body activity and hand gesture. The body activity and hand gesture are estimated using a Bayesian filter and the short-time Viterbi algorithm, which reduces the storage memory and the computational complexity. We conducted experiments in a mock apartment environment and the obtained results showed the effectiveness and accuracy of our algorithms.

In this paper, we address natural human-robot interaction (HRI) in a smart assisted living (SAIL) system for the elderly and the disabled. Two common HRI problems are studied: hand gesture recognition and daily activity recognition. For hand gesture recognition, we implemented a neural network for gesture spotting and a hierarchical hidden Markov model for context-based recognition. For daily activity recognition, a multisensor fusion scheme is developed to process motion data collected from the foot and the waist of a human subject. Experiments using a prototype wearable sensor system show the effectiveness and accuracy of our algorithms.

Nowadays, most embedded devices need to support multiple applications running concurrently. In contrast to desktop computing, very often the set of applications is known at design time and the designer needs to assure that critical applications meet their constraints in every possible use-case. In order to do this, all possible use-cases, i.e. subset of applications running simultaneously, have to be verified thoroughly. An approach to reduce the verification effort, is to perform composability analysis which has been studied for sets of applications modeled as Synchronous Dataflow Graphs. In this paper we introduce a framework that supports a more general parallel programming model based on the Kahn Process Networks Model of Computation and integrates a complete MPSoC programming environment that includes: compiler-centric analysis, performance estimation, simulation as well as mapping and scheduling of multiple applications. In our solution, composability analysis is performed on parallel traces obtained by instrumenting the application code. A case study performed on three typical embedded applications, JPEG, GSM and MPEG-2, proved the applicability of our approach.

Birds, bees, and fish often flock together in groups to find the source of food (target) based on local information. Inspired by this natural phenomenon, a flocking control algorithm is designed to coordinate the activities of multiple agents in noisy environments. Based on this algorithm, all agents can form a network and maintain connectivity. This is of great advantage for agents to exchange information. In addition, collision avoidance among agents is guaranteed in the whole process of target tracking. We show that even with noisy measurements the flocks can achieve cohesion and follow the moving target. We also investigate the stability and scalability of our algorithm. The numerical simulations are performed to demonstrate the effectiveness of the proposed algorithm.

Wireless Multimedia Sensor Networks (WMSNs) are gaining popularity among researchers over the past few years. Knowledge of the geographic locations of the sensor nodes is very important in a WMSN. In this paper we propose a new algorithm which uses the connectivity information, the estimated distance information among the sensor nodes, as well as the vision images to find the location of the sensor nodes to enable sensor calibration. We achieve this by solving an ID association problem in the WMSN. We then generate local maps for nodes in immediate vicinity and merge them together to get a global map. We demonstrate the effectiveness of our proposed approach through computer simulation.

In this paper, we propose a human activity recognition method by fusing the data from two wearable inertial sensors attached to one foot and the waist of a human subject, respectively. Our multi-sensor fusion based method combines neural networks and hidden Markov models (HMMs), and can reduce the computation load. We conducted experiments using a prototype wearable sensor system and the obtained results prove the effectiveness and the accuracy of our algorithm.

This paper presents a new approach to flocking control of a mobile sensor network to track a moving target. In our approach, the center of mass (CoM) of positions and velocities of all mobile sensors in the network (Single-CoM) or the center of mass of position and velocity of each sensor and its neighbors (Multi-CoM) is controlled to track and observe a moving target. In addition, we prove that the CoM of position and velocity exponentially converges to the moving target in free space. Based on this approach, the target is kept at the center of the sensor network. This is of great advantage for sensors to track and observe the target for recognition or identification purposes. In addition, collision-free and velocity matching among mobile sensors are guaranteed in the whole process of the target tracking. We also investigate the stability of our algorithms. The numerical simulations are performed to demonstrate the proposed approach.

In this paper, we propose a human activity recognition method by fusing the data from two wearable inertial sensors attached to one foot and the waist of a human subject, respectively. Our multi-sensor fusion based method combines neural networks and hidden Markov models (HMMs), and can reduce the computation load. We conducted experiments using a prototype wearable sensor system and the obtained results prove the effectiveness and the accuracy of our algorithm.

Target tracking is an important task in sensor networks, especially in mobile sensor networks. Flocking control is used to control a mobile sensor network to track a target. However, there are some existing problems in this control method, such as network fragmentation, loss of formation and poor tracking performance. In order to handle these problems we propose a novel approach to flocking control of a mobile sensor network to track a moving target within changing environments. In our approach, each agent can cooperatively learn the network's parameters to decide the size of network in a decentralized fashion so that the connectivity, formation and tracking performance can be improved when avoiding obstacles. In addition, to demonstrate the benefit of our approach a comparison between this approach and the existing method is given. Computer simulations are performed to demonstrate the effectiveness of the proposed approach.

In camera sensor network research, physical camera sensor network platforms with low power and low cost are needed for testing and validating algorithms. It is also necessary that the camera nodes be calibrated precisely. In this work, we first develop and evaluate a low power and low cost wireless camera sensor network platform. The camera sensor nodes in this platform transmit a grayscale image over a wireless channel to a master control station. Then we propose a simple, light-weight algorithm to perform distributed calibration of the camera sensor nodes. The camera sensor nodes use their imaging abilities in collaboration with a cooperative moving target to determine their own positions and orientations. The proposed algorithm requires simple arithmetic calculations and hence it can be realized on low power processors. This calibration algorithm has been implemented and evaluated on the developed physical camera sensor network platform.

This paper presents a low-cost, simple automated mobile platform for 3D environmental digitization. Compared with other customized 3D digitization platforms, all parts in our platform are commercial off-the-shelf. In order to build a 3D model of a salient target, we present a new viewpoint planning method for fast 3D digitization. Within a predefined accuracy, our method can determine the minimum overlap between two consecutive scanning images to speed up the digitization process. This guarantees the next scanning image can be merged to the previous one properly. The results from both simulation and experiments show the effectiveness of our viewpoint planning method. Our tests of this mobile robot system demonstrate the feasibility of 3D digitization based on a low-cost platform.

In this paper, we propose an online hand gesture recognition algorithm for a robot assisted living system. A neural network-based gesture spotting method is combined with the hierarchical hidden Markov model (HHMM) to recognize hand gestures. In the segmentation module, the neural network is used to determine whether the HHMM-based recognition module should be applied. In the recognition module, Bayesian filtering is applied to update the results considering the context constraints. We implemented the algorithm using an inertial sensor worn on a finger of the human subject. The obtained results prove the accuracy and effectiveness of our algorithm.

In the past few years, MPSoC has become the most popular solution for embedded computing. However, the challenge of programming MPSoCs also comes as the biggest side-effect of the solution. Especially, when designers have to face the legacy C code accumulated through the years, the tool support is mostly unsatisfactory. In this paper, we propose an integrated framework, MAPS, which aims at parallelizing C applications for MPSoC platforms. It extracts coarsegrained parallelism on a novel granularity level. A set of tools have been developed for the framework. We will introduce the major components and their functionalities. Two case studies will be given, which demonstrate the use of MAPS on two different kinds of applications. In both cases the proposed framework helps the programmer to extract parallelism efficiently.

This paper addresses the problem of how to navigate a miniature Crawler robot in a typical engineered structure inspection application-aircraft rivet inspection. First, a novel vision-assisted localization algorithm is developed to find the heading and position of the Crawler robot. Second, a new algorithm is developed to solve the path planning problem so that the Crawler robot can navigate through all the rivets. Experimental results validate the localization and path planning algorithms. This inspection system can be extended to other similar engineered structure inspection applications.

Tool path planning for automated manufacturing processes is a computationally complex task. This paper addresses the problem of tool path integration in the context of spray-forming processes. Tool paths for geometry-complicated parts are generated by partitioning them into individual freeform surfaces, generating the paths for each partition, and then, finally, interconnecting the paths from the different patches so as to minimize the overall path length. We model the problem as a variant of the rural postman problem (RPP), which we call open-RPP. In this paper, we present two different solutions to the open-RPP. The first solution is based on genetic algorithms and the second one is based on ant colony optimization. This paper presents and compares the results from both methods on sample data and on real-world automotive body parts. We conclude this paper with remarks about the effectiveness of our implementations and the pros and cons of each method.

We consider the problem of generating global feasible trajectories for nonholonomic mobile robots in the presence of moving obstacles. The global trajectory is composed of regional path segments, which are parametric polynomials incorporating collision avoidance criteria. Collision avoidance with moving obstacles is achieved by changing parameters of the regional trajectories, and the collision avoidance parameters are solutions to a second-order inequality and are obtainable analytically. To have a smooth global trajectory, we also develop a smooth irregular curve method to generate continuous boundary conditions to connecting regional trajectories. Steering control laws are constructed by means of differential flatness. The proposed technique works in dynamic environments where a set of global way-points are available, and the velocities of the obstacles are obtainable real time. Simulation results show the effectiveness of the methods.

Accuracy is essential to surface quality control when a range sensor is applied to measure the 3D shape of an automotive body part. A sensor's viewing pose, including location and orientation, is related to measurement accuracy. It is usually difficult to find an optimal solution by manual control of sensor viewpoints. A CAD-guided robot view planner developed previously can automatically generate viewpoints. Measurement accuracy can be satisfied in a certain range. However, the unpredictable image noises, especially in regions with low intensity contrast, cannot be compensated by the CAD-guided robot view planner. In another aspect, measurement accuracy is evaluated all over the part surface. The local accuracy of a small patch may exceed the measurement tolerance. In this paper, feedback design is applied to the CAD-guided robot sensor planning system. The feedback controller can evaluate the accuracy of obtained point clouds, identify problem regions, and generate new viewpoints. This process is recursively executed until the measurement accuracy reaches to a tolerant value. This feedback-based inspection system had been implemented in previous work to fill holes of a point cloud, which are caused by shadows and light reflections. In this paper, the feedback controller is specifically designed to improve the measurement accuracy. Experimental results show the success of applying this feedback system for dimensional inspection of an automotive body part.

In this paper, we define a mobile self-localization (MSL) problem for sparse mobile sensor networks, and propose an algorithm named mobility assisted MDS-MAP(P), based on multi-dimensional scaling (MDS) for solving the problem. For sparse sensor networks, all the existing localization algorithms fail to work properly due to the lack of distance or connectivity data to uniquely calculate the geo-locations. In MSL, we use mobile sensors to add extra distance constraints to a sparse network, by moving the mobile sensors in the area of deployment and recording distances to neighbors at some intermediate locations. MSL can also be used for localizing and tracking mobile objects in a robotic or body sensor network. Experiments and evaluations of the new algorithm are provided.

In this paper, a dynamic localization algorithm is proposed for the tracking of mobile subjects in a wireless ad hoc network environment. This algorithm integrates a dynamic multidimensional scaling (MDS) method and the dead reckoning method enabled by a wearable position tracking (WePoST) system through sensor fusion technique. The essential idea of the dynamic MDS (DMDS) is to reduce the localization error by increasing the network density and connectivity through the addition of virtual nodes. Sensor fusion technique is used to improve the consistency of the dynamic MDS algorithm by integrating the knowledge of short distance displacement. Tests done in simulation verify the proposed dynamic localization algorithm.

This paper introduces a novel robotic range sensor planning system, which is developed for 3D dimensional inspection of automotive body parts. For active, triangulation-based range sensors, shadow and reflection causes problems when measure a metal part with glossy and discontinuous surface. To solve these problems, a feedback based dynamic view planning system is proposed, which not only generate viewpoints from a CAD model of a part, but also recursively add viewpoints according to the measured information. The planning process stops only if the obtained point clouds meet the pre-determined requirements. General framework of the system is introduced, and the experimental results are also presented

Structural health monitoring and inspection is very important for many civil, mechanical and aerospace systems. It is a critical step in maintaining and improving the structural integrity of these systems. In this paper, we propose to develop an automated, intelligent inspection system for these engineered structures, which employs a team of intelligent climbing robots and a command robot to collaboratively carry out the inspection task. To support autonomous navigation, a Miniature Active Camera (MaCam) module is designed, which can be used in the pose calibration of the robot. The robot pose error model is introduced. Based on that, the path planning problem for a single robot is studied and a hierarchical algorithm is developed to generate a path that satisfies the navigation requirements of the robot. Both the error model and the path planning algorithm is verified in experiments.

A new sensor network architecture, active sensor network (ASN), is proposed in this paper. This architecture integrates multiple sensor network-friendly mobile robots into a traditional sensor network. Therefore a closed-loop, dynamic adaptive sensor network is formed, which has many desired merits. In this paper, the distributed sensor node localization and the service set partition among multiple micro mobile robots are studied. For the node localization, a potential-based robot area partition algorithm and a distributed localization algorithm are developed respectively. For service set partition, a load-balanced partitioning algorithm is developed, which adapts to the number of active nodes in the field. The proposed algorithms are verified through simulations

This paper describes our efforts in building an experimental platform to conduct research on cooperative driving in intelligent transportation systems (ITS). A miniature vehicle is developed based on the COTS-BOTS robot and an overhead vision system is used to localize the vehicles. The single lane tracking control algorithm is described first. To control multiple vehicles to drive through an intersection without collision, a distributed cooperative driving algorithm is developed, which is based on velocity planning using search algorithms in the corresponding coordination diagram. Experimental results verify our proposed approach and we believe this platform can serve as a cost-effective testbed to study vehicle-to-vehicle communication based cooperative driving in intelligent transportation systems

Point cloud registration and sensor calibration are two critical technical issues concerning robot-mounted, area sensor systems. Iterative closest point (ICP)-based algorithms developed in the past are commonly used for point cloud registration. However, due to its least squared fitting nature, registration quality depends on how closely the measured part matches its nominal definition, typical in the form of a CAD model in modern times. To eliminate the dependency of registration quality on part closeness to the CAD model, we present, in this paper, a more robust approach based in a series of coordinate transformations. Geometric features and surface gradients are accounted for to improve the registration performance. To achieve robot/sensor hand-eye calibration, an ICP-based method is used. The reason is that this calibration step typically utilizes standard parts or gauges machined for the purpose of calibration, as such they are known shapes that match their CAD models with much tighter tolerances. This offers us an unique opportunity to apply an ICPbased tool to find the transformation matrix from the robot end effector to an area sensor mounted onto it. The discussed method was successfully implemented and tested in a feedback-based, robot-mounted area sensor system developed for manufacturing quality control of 3D freeform surfaces

Spray forming is a new manufacturing process. The automated tool planning for this process is a challenging problem, especially for geometry-complicated parts consisting of multiple freeform surfaces. We have developed a tool path planning system which can automatically generate tool plants. This paper discusses the path integration problem, i.e, how to connect the paths from different surface patches so that the overall path length is minimized. An Ant Colony Optimization (ACO)-based path integration algorihm is developed to solve this problem. Experimental tests are carried out on sample data as well as real world automotive body parts. The results proved to be better (by up to 10%) compared with those of a Genetic Algorithm (GA)-based solution we developed before.

For the aircraft structure inspection, this paper introduces a micro biped robot with inspection probe and wireless vision, analyzes the robot locomotion modes and dynamic models, and studies the motion control algorithm. Considering the movement flexibility caused by five degrees of freedom, a hierarchy structure is presented for the robot motion control system. For the long distance locating problem, a relay locating approach is presented to solve robot locating and to estimate the orientation of the robot by using vision and distance information from encoder and CAD model. The movement orientation can be adjusted rivet by rivet in the inspection process. The experimental results show that the control algorithm works well, and orientation estimation algorithm provides an acceptable orientation precision for continuous rivet inspection.

This paper addresses the path planning problem of a mobile sensor, which is a micro Crawler robot equipped with a Eddy Current probe, for aircraft rivet inspection. Due to the specific movement characteristic of the Crawler robot, the path, or the rivet sequence should enable the Crawler robot to realize the localization-and-reposition process. Therefore a final path is subject to the following three requirements: i) the distance between any two consecutive rivets on the path should be less than a threshold distance; ii) the number of turns should be minimized and iii) the overall distance should be minimized. In this paper, a novel algorithm is developed to generate the required path. This algorithm first identifies the minimum set of line segments partitioning all the rivets and then connects the line segments to minimize the overall distance. Simulation results are provided to validate the proposed algorithm.

Automatic chopper gun trajectory generation for spray forming is highly desirable for today's automotive manufacturing. Generating chopper gun trajectories for free-form surfaces to satisfy the given requirements is still highly challenging due to the complex geometry of free-form surfaces and the spray gun model. A CAD-guided chopper gun trajectory generation system for both uniform and nonuniform material distribution of free-form surfaces has been developed in our previous work. A material distribution model is also presented. To verify the developed algorithms, experiments were performed. In this paper, the experimental results are presented and compared with the simulation results. The results demonstrate that the developed trajectory generation system can be applied to generate trajectories for free-form surfaces such that the material distribution requirements can be satisfied. Also, the material distribution model can be used to compute the material distribution for free-form surfaces. This trajectory generation method can also be applied to generate trajectories for many other CAD-guided robot trajectory planning applications.

Information sharing through explicit communication is necessary in many multi-robot applications in order to achieve effective decision making. However, unnecessary large volumes of communication data usually lead to time delay and energy waste. This paper addresses the problem of how to efficiently synchronize the map (or explored area) among multiple robots when they carry out cooperative area exploration or coverage. In this application, multiple robots need exchange map information in order to minimize the repeated exploration or coverage. When a connected ad hoc network can not be maintained, the map synchronization problem becomes more challenging. In this paper, a sequence number based map representation scheme and an effective map update tracking scheme are proposed. Based on them, an algorithm is developed to reduce the volumes of map exchange when robot subnetworks merge. Simulation results validate this algorithm.

Spray forming is an emerging manufacturing process. The automated tool planning for this process is a nontrivial problem, especially for geometry-complicated parts consisting of multiple freeform surfaces. Existing tool planning approaches are not able to deal with this kind of compound surface. This paper proposes a tool-path planning approach which optimizes the tool motion performance and the thickness uniformity. There are two steps in this approach. The first step partitions the part surface into flat patches based on the topology and normal directions. The second step determines the tool movement patterns and the sweeping directions for each flat patch. Based on the above two steps, optimal tool paths can be calculated. Experimental tests are carried out on automotive body parts and the results validate the proposed approach. Note to Practitioners-This paper was motivated by the problem of automatically planning tool paths for spray forming using Programmable Powdered Preforming Process (P4) technology. However, the proposed approach can be applied to other surface manufacturing applications such as spray painting, spray cleaning, rapid tooling, etc. Existing tool planning approaches are not able to handle complicated, multi-patch surfaces. This paper proposes a methodology to partition complicated surfaces into easy-to-handle patches and generate tool paths with optimized thickness uniformity and tool motion performance. We tested the approach using simulation on sample automotive body parts and proved its feasibility. However, this approach requires that the parts to be sprayed belong to the sheet-metal type so that the part geometry can be analyzed on a plane. In our future research, we will run physical tests on actual parts and investigate the deposition effects on the thickness uniformity.

Architecture description languages (ADL) have been established to aid the design of application-specific instruction-set processors (ASIP). Their main contribution is the automatic generation of a software toolkit, including C compiler, assembler, linker, and instruction-set simulator. Hence, the challenge in the design of such ADLs is to unambiguously capture the architectural information required for the toolkit generation in a single model. This is particularly difficult for C compiler and simulator, as both require information about the instructions' semantics, however, while the C compiler needs to know what an instructions does, the simulator needs to know how. Existing ADLs solve this problem by either introducing redundancy or by limiting the language's flexibility.This paper presents a novel, mixed-level approach for ADL-based instruction-set description, which offers maximum flexibility while preventing from inconsistencies. Moreover, it enables capturing instruction- and cycle-accurate descriptions in a single model. The feasibility and design efficiency of our approach is demonstrated with a number of contemporary, real-world processor architectures.

Automated chopper gun trajectory planning for spray forming is highly desirable for today's automotive manufacturing. Generating chopper gun trajectories for free-form surfaces to satisfy thickness requirements is still highly challenging due to the complexity of problems. Automated trajectory planning for constant material distribution has been widely studied. However, achieving a variational (non-uniform) material distribution has not been addressed. A CAD-guided chopper gun trajectory planning system with non-uniform material distribution for free-form surfaces is presented. A multi-objective constrained optimization problem is formulated. The simulation results have shown that this system achieves performance required by production applications. This trajectory planning algorithm can also be applied to generate trajectories for other CAD-guided robot trajectory planning applications.

Spray forming is an emerging manufacturing process. The automated tool planning for this process is a nontrivial problem, especially for geometry-complicated parts consisting of multiple freeform surfaces. Existing tool planning approaches are not able to deal with this kind of compound surfaces. This paper proposes a tool path planning approach which considers the tool motion performance and the thickness uniformity. There are two steps in this approach. The first step partitions the part surface into flat patches based on its topology and normal directions. The second step determines the tool movement patterns and the sweeping directions for each flat patch. Based on that, optimal tool paths can be calculated. Experimental tests are carried out on automotive body parts and the results validate the proposed approach.

This paper presents a distributed model for cooperative multiple mobile robot systems. In a multiple robot system, each mobile robot has sensing, computation and communication capabilities. The mobile robots spread out across certain area and share sensory information through an ad hoc wireless network. The multiple mobile robot system is therefore a mobile sensor network. In this paper, Voronoi diagram and Delaunay triangulation are introduced to model the area coverage and cooperation of mobile sensor networks. Based on the model, this paper discusses a fault tolerant algorithm for autonomous deployment of the mobile robots. The algorithm enables the system to reconfigure itself such that the area covered by the system can be enlarged. The proposed formation control algorithm allows the mobile sensor network to track moving target and sweep a larger area along specified paths.

This paper presents a fuzzy system approach that incorporates sensing, control and planning to enable micro wall climbing robots navigating in unstructured environments. Based on a fuzzy multi-sensor data fusion scheme, a novel strategy for integrating task scheduling, sensing, planning and real-time execution is proposed so that task scheduling, action planning and motion control can be treated in a unified framework, and the design of task synchronization becomes much easier. A hybrid method is employed to create robot's gaits by switching the robot between continuous motion modes with the help of a finite state machine driven by the signals from robot sensors. For a highly non-linear system as the micro wall climbing robot, a fuzzy motion controller is designed to improve the robot's performance at different situations. Experimental results prove the validity of the proposed method.

This paper proposes a reliable and efficient multi-robot coordination algorithm to accomplish an area exploration task given that the communication range of each robot is limited. This algorithm is based on a distributed bidding model to coordinate the movement of multiple robots. Two measures are developed to accommodate the limited-range communications. First, the distances between robots are considered in the bidding algorithm so that the robots tend to stay close to each other. Second, a novel coding mechanism is introduced in the map representation to reduce the exchanged data volume when new communication links are formed. Simulation results show the effectiveness of the use of the nearness measure in the coordination algorithm as well as the new coding mechanism to reduce the amount of map data exchanged. By handling the limited communication range we can make the coordination algorithms more practical in multi-robot applications.

The timed Petri net model introduces the possibility of applying a set of mathematical results, mainly based on the use of max-plus algebra, for performance analysis. This paper aims to build a new scheduling method for a flexible manufacturing work-cell by merging the timed Petri net model and max-plus algebra. The results can be computed as functions of a certain set of decision parameters. These functions can be used to schedule, plan and control the flexible manufacturing work-cell, so that it can real-timely adapt itself to machine faults.

Surface manufacturing is widely used in industry. Automatic CAD-guided tool planning has many applications in surface manufacturing, such as spray painting, spray forming, rapid tooling, cleaning and polishing. According to the material quantity requirements, these tasks can be categorized into two groups: the material uniformity problem and coverage problem. A general framework is developed to automatically generate trajectories of a free-form surface for these tasks. A given task is first transformed into one of the groups. Based on the CAD model of a free-form surface, constraints and tool model, a trajectory for a free-form surface is generated. Velocity optimization is discussed to optimize the material quantity. Simulations are performed to verify the developed framework. This framework can also be extended to other applications.

This paper presents an analysis and design method for human/robot integrated systems, especially fro the coordination of human and robot formations based on a group of distributed mobile manipulators. The key for the human/robot integrated system is to create a common motion reference that can be understood by both human and the robots in the formation. First, the perceptive reference frame is introduced and the characteristics of perceptive frame are compared with time based reference frame. Next, the stability of the system based on perceptive reference frame is investigated. The applications of perceptive reference frame to multi-agents coordination in a formation, human/mobile manipulators coordination are then discussed. Based on the perceptive reference frame, the human can be naturally integrated into the robot formation. Human intelligence can therefore be integrated with the mobility and dexterous manipulation capability of the mobile manipulators to undertake complex tasks. The robot formation is therefore able to reconfigure and thus cope with unexpected events. Experiments have been used to verify the theoretical results.

Additive manufacturing processes such as spray coating, spray painting and rapid tooling important steps in many products' life cycle. Robotic manipulators are widely adopted to carry out these processes. If done by human operators, the tool planning for these applications is usually time-consuming and the generated tool plans are prone to inaccuracy and errors. This research develops a fully-automated, CAD-guided tool planning system which eliminates the human involvement. Meanwhile, this system can generate optimized tool plans in the sense of robot motion performance. The critical part of this tool planning system is the partitioning of part surfaces into multiple easy-to-handle patches. In this paper, a decomposition-based approach is developed, which models the surface partitioning problem in geometric domain as an integer programming problem in algebraic domain. Experimental tests and evaluation carried out on automotive parts validate this new approach.

Automatic trajectory generation for spray painting is highly desirable for today's automotive manufacturing. Generating paint gun trajectories for free-form surfaces to satisfy paint thickness requirements is still highly challenging due to the complex geometry of free-from surfaces. In this paper, a CAD-guided paint gun trajectory generation system for free-form surfaces has been developed. A paint thickness verification method is also provided to verify the generated trajectories. The results of experiments and simulations show that the trajectory generation system achieves satisfactory performance. This trajectory generation system can also be applied to generate trajectories for many other CAD-guided robot trajectory planning applications.

This paper addresses the robot path planning problem for automotive part inspection using structured light method. This problem is rendered as a traveling salesman problem (TSP). A new approach is developed to solve the TSP into its sub-optimality quickly. Instead of solving a large size TSP, this approach utilizes the clustered nature of the viewpoints and converts the TSP into a clustered traveling salesman problem (CTSP). A new algorithm, which favors the inter-group paths, is proposed to solve the CTSP. Experimental results on various automotive parts validate the new algorithm.

A CAD-guided camera planning system for dimensional inspection of automotive parts has been developed. This system utilizes the CAD information of inspected parts and the camera model to plan camera viewpoints that satisfy certain task constraints. To overcome some limitation inherited in the existing method, a new CAD data representation is adopted to combine the advantages of parametric surface and triangulation. A graph-based surface merging algorithm is proposed to reduce the number of flat patches, and eventually, the number of viewpoints. Tests on different real part models show that the surface merging algorithm is effective. This algorithm can also benefit many other CAD-guided robot path planning applications.

Sensor planning is critical in many inspection systems. In typical part dimensional inspection systems, eye-in-hand robots are usually adopted. One practical problem that needs to be addressed is that the viewpoints, or correspondingly, the robot hand-tip positions and orientations, should be within the robot's reachability space. Furthermore, it is desirable that the robot can reach these viewpoints "comfortably". In this sense the robot kinematics constraint should be considered in sensor planning. In the paper, an approach is developed to integrate the kinematics constraint into sensor planning. This approach models the problem as an integer programming problem. With this approach, the computation burden of the robot placement problem can be avoided. Experiment and simulation results demonstrate the effectiveness of our approach.

Structured light is one of the well-known methods in part dimensional inspection that have been successfully employed in various applications in the past decades. In this method, the positioning of the camera is very critical, which affects the accuracy and efficiency of the whole inspection system. Here we develop a CAD-guided camera positioning system to aid the 3-D part inspection. The geometric information in the CAD model of the inspected part and the camera model are used to plan camera configurations that satisfy certain task constraints. The overall system we propose can be applied to the 3-D inspection of parts with free-form surfaces, such as automobile door panels. Experiments on different parts show satisfying results.

In this paper, a CAD-guided robot path generator is developed for the spray painting of compound surfaces commonly seen in automotive manufacturing. Instead of widely used parametric representation of surfaces, a planar facet scheme is used to approximate the painting surfaces. As a result, a new combinatorial gun path planning algorithm is proposed. In this algorithm, big patches are formed by stitching small surfaces together. Therefore, the path planning is solved based on the global characteristics of tire part, and the resulting spray gun paths are well-behaved in the sense of time, coverage, and wastage. The proposed algorithm has been implemented and tested using ROBCAD/sup TM//Paint.