The COVID-19 pandemic has caused a global public health crisis, leading to increased costs for operating essential quarantine facilities and risks of infection for medical staff. To address these issues, we have developed a non-contact delivery robot, UTD-pro, that can deliver food and supplies to patients in an isolated residential treatment center. In case of emergency situations such as communication failure with the control system or malfunction of specific modules, the noncontact delivery robot should be able to perform commands robustly or return home. Otherwise, as before, medical staffs may have to enter the high-risk environment again. In this paper, we propose a flexible and robust remote control system and a robot task management system that enables the robot to execute commands correctly. Our robot task manager allows for autonomous task execution with a single command from the remote control system. Each robot task operates independently, allowing users to change task plans flexibly during robot operation. The sensor data communication and task execution communication processes are also independent, preventing any issue in one process from affecting the other. By applying our system to the robot, we conducted a long-term delivery experiment for a total of 7.723km for 79 days, and achieved 125 successes out of 149 trials. This experiment proves that our system can lead to stable delivery processes and contribute to the high reliability of the control system and the robot task manager.

A social robot interacts with humans based on social intelligence, for which related applications are being developed across diverse fields to be increasingly integrated in modern society. In this regard, social intelligence and interaction are the keywords of a social robot. Social intelligence refers to the ability to control interactions or thoughts and feelings of relationships with other people; primal empathy, which is the ability to empathize by perceiving emotional signals, among the components of social intelligence was applied to the robot in this study. We proposed that the empathic ability of a social robot can be improved if the social robot can create facial expressions based on the emotional state of a user. Moreover, we suggested a framework of facial expressions for robots. These facial expressions can be repeatedly used in various social robot platforms to achieve such a strategy.

For a service robot to assist humans, it should interact with objects of varying sizes and shapes existing in an indoor environment. 3D object detection must be preceded to achieve this goal since it provides the robot with the ability to perceive visual information. Most of the existing methods are anchor-based and predict the bounding box close to the ground truth among multiple candidates. However, it is complex to compute Intersection over Union (IoU) and Non-Maximum Suppression (NMS) per each anchor box. Therefore, we propose keypoint-based monocular 3D object detection, where each object's center location is only needed for reproducing predicted 3D bounding box without extra computation of the anchor boxes. Our 3D object detection also works well even if images are rotated corresponding to the robot's head movement. To properly train our network, the object center is based on a projected 3D location instead of 2D to take advantage of the exact center position of the object. Furthermore, we apply data augmentation using a perspective transformation. The method facilitates adding a small perturbation to the camera rotation angle randomly. We use the SUN RGB-D dataset, which has images taken indoor scenes with camera rotations for training and test set. Our approach particularly shows that the errors of object center location based on a single image reduce 15.4% and 24.2%, respectively, compared to the method without data augmentation.

We propose a method designed to push the frontiers of unconstrained face recognition in the wild with an emphasis on extreme out-of-plane pose variations. Existing methods either expect a single model to learn pose invariance by training on massive amounts of data or else normalize images by aligning faces to a single frontal pose. Contrary to these, our method is designed to explicitly tackle pose variations. Our proposed Pose-Aware Models (PAM) process a face image using several pose-specific, deep convolutional neural networks (CNN). 3D rendering is used to synthesize multiple face poses from input images to both train these models and to provide additional robustness to pose variations at test time. Our paper presents an extensive analysis of the IARPA Janus Benchmark A (IJB-A), evaluating the effects that landmark detection accuracy, CNN layer selection, and pose model selection all have on the performance of the recognition pipeline. It further provides comparative evaluations on IJB-A and the PIPA dataset. These tests show that our approach outperforms existing methods, even surprisingly matching the accuracy of methods that were specifically fine-tuned to the target dataset. Parts of this work previously appeared in [1] and [2].

In recent years face recognition has made extraordinary leaps, yet unconstrained video-based face identification in the wild remains an open and interesting problem. Videos, unlike still-images, offer a myriad of data for face modeling, sampling, and recognition, but, on the other hand, contain low-quality frames and motion blur. A key component in video-based face recognition is the way in which faces are associated through the video sequence before being used for recognition. In this paper, we present a video-based face recognition method taking advantage of face and body association (FBA). To track and associate subjects that appear across frames in multiple shots, we solve a data association problem using both face and body appearance. The final recovered track is then used to build a face representation for recognition. We evaluate our FBA method for video-based face recognition on a challenging dataset. Our experiments show up to 5% improvement in the identification rate over the state-of-the-art.

This paper concerns the problem of facial landmark detection. We provide a unique new analysis of the features produced at intermediate layers of a convolutional neural network (CNN) trained to regress facial landmark coordinates. This analysis shows that while being processed by the CNN, face images can be partitioned in an unsupervised manner into subsets containing faces in similar poses (i.e., 3D views) and facial properties (e.g., presence or absence of eye-wear). Based on this finding, we describe a novel CNN architecture, specialized to regress the facial landmark coordinates of faces in specific poses and appearances. To address the shortage of training data, particularly in extreme profile poses, we additionally present data augmentation techniques designed to provide sufficient training examples for each of these specialized sub-networks. The proposed Tweaked CNN (TCNN) architecture is shown to outperform existing landmark detection methods in an extensive battery of tests on the AFW, ALFW, and 300W benchmarks. Finally, to promote reproducibility of our results, we make code and trained models publicly available through our project webpage.

A key to successful face recognition is accurate and reliable face alignment using automatically-detected facial landmarks. Given this strong dependency between face recognition and facial landmark detection, robust face recognition requires knowledge of when the facial landmark detection algorithm succeeds and when it fails. Facial landmark confidence represents this measure of success. In this paper, we propose two methods to measure landmark detection confidence: local confidence based on local predictors of each facial landmark, and global confidence based on a 3D rendered face model. A score fusion approach is also introduced to integrate these two confidences effectively. We evaluate both confidence metrics on two datasets for face recognition: JANUS CS2 and IJB-A datasets. Our experiments show up to 9% improvements when face recognition algorithm integrates the local-global confidence metrics.

A framework to generate a human-like arm motion of a humanoid robot using an Evolutionary Algorithm (EA)-based imitation learning is proposed. The framework consists of two processes, imitation learning of human arm motions and real-time generating of a human-like arm motion using the motion database evolved in the learning process. The imitation learning builds the database for the humanoid robot that is initially converted from human motion capture data and then evolved using a genetic operator based on a Principal Component Analysis (PCA) in an evolutionary algorithm. This evolution process also considers the minimizing of joint torques in the robot. The database is then used to generate humanoid robot's arm motions in real-time, which look like human's and require minimal torques. The framework is examined for humanoid robot to reach its arms for catching a ball. Additionally, the inverse kinematics problem to determine the final posture of 6-DOF robot arm with a waist for the task of catching a ball, is proposed.