Learning visual representation of high quality is essential for image classification. Recently, a series of contrastive representation learning methods have achieved preeminent success. Particularly, SupCon [18] outperformed the dominant methods based on cross-entropy loss in representation learning. However, we notice that there could be potential ethical risks in supervised contrastive learning. In this paper, we for the first time analyze unfairness caused by supervised contrastive learning and propose a new Fair Supervised Contrastive Loss (FSCL) for fair visual representation learning. Inheriting the philosophy of supervised contrastive learning, it encourages representation of the same class to be closer to each other than that of different classes, while ensuring fairness by penalizing the inclusion of sensitive attribute information in representation. In addition, we introduce a group-wise normalization to diminish the disparities of intra-group compactness and inter-class separability between demographic groups that arouse unfair classification. Through extensive experiments on CelebA and UTK Face, we validate that the proposed method significantly outperforms SupCon and existing state-of-the-art methods in terms of the trade-off between top-l accuracy and fairness. Moreover, our method is robust to the intensity of data bias and effectively works in incomplete supervised settings. Our code is available at https://github.com/sungho-CoolG/FSCL.

The development of artificial intelligence has led to significant progress in activity detection; however, a proper evaluation of the activity detection performance is not guaranteed and is still lacking in real-life environments. In this study, to verify the stability and usefulness of activity detection in a real-world setting, we analyzed the activity detection performance for 40 elderly people using a human care robot.

Although AI systems archive a great success in various societal fields, there still exists a challengeable issue of outputting discriminatory results with respect to protected attributes (e.g., gender and age). The popular approach to solving the issue is to remove protected attribute information in the decision process. However, this approach has a limitation that beneficial information for target tasks may also be eliminated. To overcome the limitation, we propose Fairness-aware Disentangling Variational Auto-Encoder (FD-VAE) that disentangles data representation into three subspaces: 1) Target Attribute Latent (TAL), 2) Protected Attribute Latent (PAL), 3) Mutual Attribute Latent (MAL). On top of that, we propose a decorrelation loss that aligns the overall information into each subspace, instead of removing the protected attribute information. After learning the representation, we re-encode MAL to include only target information and combine it with TAL to perform downstream tasks. In our experiments on CelebA and UTK Face datasets, we show that the proposed method mitigates unfairness in facial attribute classification tasks with respect to gender and age. Ours outperforms previous methods by large margins on two standard fairness metrics, equal opportunity and equalized odds.

With great research advances on Brain-Computer-Interface (BCI) systems, Electroencephalography (EEG) based driver fatigue state classification models have shown its effectiveness. However, EEG signals contain large differences between individuals, making it hard to build a unified model among individuals. In this paper, we propose a subject- independent EEG-based driver fatigue state (i.e., awake, tired, and drowsy) classification model that mitigates a performance gap between subjects. To this end, we exploit an adversarial training strategy to make our classification model misclassify the subject labels. Besides, we propose an Intersubject Feature Distance Minimization (IFDM) method that minimizes the Wasserstein distance between two different subject groups of the same class to reduce the individual performance discrepancy. Our method is also designed to enable training even if the subject labels are not sufficiently included in the EEG dataset. To demonstrate the ability of the proposed method, we conduct a drowsiness classification task on a publicly available SEED-VIG dataset. The experimental results show our model achieves the highest accuracy and the lowest individual performance variability.

Deep learning, based on which many modern algorithms operate, is well known to be data-hungry. In particular, the datasets appropriate for the intended application are difficult to obtain. To cope with this situation, we introduce a new dataset called ETRI-Activity3D, focusing on the daily activities of the elderly in robot-view. The major characteristics of the new dataset are as follows: 1) practical action categories that are selected from the close observation of the daily lives of the elderly; 2) realistic data collection, which reflects the robot's working environment and service situations; and 3) a large-scale dataset that overcomes the limitations of the current 3D activity analysis benchmark datasets. The proposed dataset contains 112,620 samples including RGB videos, depth maps, and skeleton sequences. During the data acquisition, 100 subjects were asked to perform 55 daily activities. Additionally, we propose a novel network called four-stream adaptive CNN (FSA-CNN). The proposed FSA-CNN has three main properties: robustness to spatio-temporal variations, input-adaptive activation function, and extension of the conventional two-stream approach. In the experiment section, we confirmed the superiority of the proposed FSA-CNN using NTU RGB+D and ETRI-Activity3D. Further, the domain difference between both groups of age was verified experimentally. Finally, the extension of FSA-CNN to deal with the multimodal data was investigated.

This paper describes the reason that perception technology in HRI has not given high performance enough to be used for commercial service robots. As a practical solution for better performance of perception technology, we propose a perception framework with a dedicated perception engine called a perception demon. The demon in the proposed framework constantly collects evidences and analyses them better by combining various types of individual perception components. The proposed framework enables robot makers to easily get more reliable information on humans without concerns about optimizing perception components to their robots.

In everyday lives, so many events happen around us. Although not all of them are important to us, we have to process them all to be able to isolate the significant events. Consciousness is a constant awareness of the environment, which can be acquired only by such an exhaustive approach. As an autonomous agent, a robot is also required to be conscious, which is not yet properly realized. In this paper, we discuss the strategy to organize various perception technologies to achieve the consciousness of the robots. Especially, an independent process that constantly monitors its surroundings without the intervention of higher processes is proposed. This process, which is called a perception daemon, is discriminated from the traditional component-based approach, where individual perception technology is provided as a passive function so that the service applications should call whenever it is necessary.

Robotic person-following is an essential component for natural human robot interaction. To follow a person, the robot should track the target person robustly and in real time. Object tracking algorithms in the computer vision field typically require abundant features and heavy computing power, and thus cannot be directly applied to person-following robots due to the problems arising in practical robotic environments. This paper proposes a robotic person-tracking algorithm based on modified multiple instance learning. In order to resolve the problems raised by the rearward view of the target person, the tracker is modified to be guided by color histogram back-projection. Additionally, the search area model is modified from circle to ellipse and the number of features is reduced so that the tracker should adapt the robotic environment in real-time. The algorithm is validated through system integration and experiments.

Automatic facial expression recognition is an important technique for interaction between humans and machines such as robots or computers. In particular, pose invariant facial expression recognition is needed in an automatic facial expression system because frontal faces are not always visible in real situations. The present paper introduces a multi-view method for recognizing facial expressions using a parametric kernel eigenspace method based on class features (pKEMC). We first describe pKEMC that finds the manifold of data patterns in each class on a non-linear discriminant subspace for separating multiple classes. Then, we apply pKEMC for pose-invariant facial expression recognition. We also utilize facial-component-based representation to improve the robustness to pose variation. We carried out the validation of our method on a Multi-PIE database. The results show that our method has high discrimination accuracy and provides an effective means to recognize multi-view facial expressions.

The paper presents the person following method with obstacle avoidance. The robot is equipped with RGB camera and laser range finder. The method is based on the target person tracking using multi-layered mean shift and three force fields based obstacle avoidance. To validate the method, we applied this method on the robot platform. The experimental result and analysis are presented in the experiment.

The fully automatic face identification on the robot system usually meet challenging environments due to the motion blur, illumination change, and erroneous face normalization caused by imperfect face detection. In the face identification system, multiple images of same individual in gallery set can be used as useful sources for good identification. This paper introduces the face identification system using efficient comparison method between gallery and probe set using HOG features. The comparison method based on blocks makes possible that the system could effectively use the limited registration images in the classification step. The whole process is validated on ETRI-HRI Database acquired in uncontrolled environment.

Recently, some 2D based algorithms were proposed for image classification. In this paper, we propose two-dimensional logistic regression for two-class problem. This method has an advantage of memory saving and better performance than other 2D based approaches. The experimental results show two dimensional logistic regression is superior to one dimensional logistic regression and other popular two-dimensional methods.

User recognition is one of the most fundamental functionalities for intelligent service robots. However, in robot applications, the conditions are far severer compared to the traditional biometric security systems. The robots should be able to recognize users non-intrusively, which confines the available biometric features to face and voice. Also, the robots are expected to recognize users from relatively afar, which inevitably deteriorates the accuracy of each recognition module. In this paper, we tried to improve the overall accuracy by integrating the evidences issued by independently developed face and speaker recognition modules. Each recognition module exhibits different statistical characteristics in representing its confidence of the recognition. Therefore, it is essential to transform the evidences to a normalized form to integrate the results. This paper introduces a novel approach to integrate mutually independent multiple evidences to achieve an improved performance. Typical approach to this problem is to model the statistical characteristics of the evidences by well-known parametric form such as Gaussian. Using Mahalanobis distance is a good example. However, the characteristics of the evidences often do not fit into the parametric models, which results in performance degradation. To overcome this problem, we adopted a discrete PDF that can model the statistical characteristics as it is. To confirm the validity of the proposed method, we used a multi-modal database that consists of 10 registered users and 550 probe data. Each probe data contains face image and voice signal. Face and speaker recognition modules are applied to generate respective evidences. The experiment showed an improvement of 11.27% in accuracy compared to the individual recognizers, which is 2.72% better than the traditional Mahalanobis distance approach.

This paper describes a new robot photographer system which can interact with people. The goal of this research is to make the system act like a human photographer. This system is based on a mobile robot having capabilities of wireless communication and stereo vision. It recognizes waving hands of people, moves toward them, and takes pictures with designated compositions. The pictures are transmitted to personal computers over the wireless network. In comparison with previous researches, the most unique things of this system are human interaction and user preference. To realize these properties, this new robot photographer system is considered optical property of lens and applied interesting vision algorithm which was never tried before for previous robot photographer systems

In this video, we present a scenario for developing vision-based Human-Robot Interaction (vHRI) components for Ubiquitous Robot Companion (URC) Intelligent Service Robots, which exploit strong Information Technology (IT) infrastructure. Here the URC means that it will provide the necessary services at any time and place to meet the user's requirements. Thus, it combines the network function with the current concept of a robot in order to enhance mobility and human interface. The vHRI components used in this video consist of caller identification, face recognition/verification, and gesture recognition. Firstly we perform face verification and recognition for the imposter and a member of family from face images of two users, respectively. After that, we use gesture recognition to provide the selection of TV channels under noise or long-distance environments. Finally, we demonstrate the usefulness through vision-based scenario shown from video. The robot platform used in this video is WEVER, which is a URC intelligent service robot developed at Intelligent Robot Research Division in Electronics and Telecommunication Research Institute (ETRI).

In this paper, an algorithm for human head detection over a distance exceeding 2.5 m between a camera and an object is described. This algorithm is used for the control of a robot, which has the additional limits of a moving camera, moving objects, various face orientations, and unfixed illuminations. With these circumstances, only the assumption that human head and body contours have an omega (Omega) shape is made. In order to separate the background from these omega shapes, the three basic features of gray, color and edge are used, and combined. The skin color cue is very useful when the image stream is frontal and has large face regions, and additionally has no background objects similar to the skin color. The gray cue is also important when captured faces have a lower gray level than background objects. The edge cue is helpful when captured background objects have similar gray levels and colors to those of a head, but can be discriminated by edges. Since these three methods have roughly orthogonal failure results, they serve to complement each other. The next step is a splitting method between the head and body region using the proposed method. The final step is an ellipse fitting and a head region verification algorithm. The results of this algorithm provide robustness for head rotation, illumination changing, and variable head sizes. Furthermore, it is possible to carry out real time processing

The ability to recognize people is positively necessary for interaction between robots and people. In this paper, we introduce intelligent security system that is able to recognize, track and remember user's face. System uses moving camera that has pan/tilt and zoom function like robot's eyes. In order to construct system, we propose the face identification method based on combining multiple features and applying multiple matching algorithms. Also, we address the benefits of multiple instances and show that identification ability can be improved by using proper combination strategy

The user authentication problem is discussed in the context of robot environments. Although authentication is indispensable for useful robot services, the traditional approaches that have been applied in strictly regulated environments cannot be applied as they are when both the robot and humans are freely moving around. In this paper, a new authentication system redesigned to be adapted to robot environments is described. In the proposed system, semi-biometrics and user tracking are incorporated to cope with the limited applicability of traditional authentication techniques. Semi-biometrics indicates a set of features useful for discriminating persons, but only in interested group of persons and in interested frame of time. As a representative semi-biometric feature, body height and color characteristics of the clothes are investigated. Especially, a novel method to measure body height with single camera is proposed. Also by incorporating tracking functionality, the system could maintain the user status information continuously, which is useful not only for recognition but also for finding a designated person.

The paper proposes a novel face verification algorithm based on multiple feature combination and a support vector machine. The main issue in face verification is to deal with the variability in appearance. It seems difficult to solve this issue by using a single feature. Therefore, combination of mutually complementary features is necessary to cope with various changes in appearance. From this point of view, we describe feature extraction approaches based on multiple principal component analysis and edge distribution. These features are projected on a new intra-person/extra-person similarity space that consists of several simple similarity measures, and are finally evaluated by a support vector machine supervisor. From the experiments on a realistic and large database, an equal error rate of 0.029 is achieved, which is a sufficiently practical level for many real-world applications.

This paper proposes a novel face verification algorithm based on multiple feature combination and a support vector machine. The main issue in face verification is to deal with the variability in appearance. It seems difficult to solve this issue using a single feature. Therefore, a combination of mutually complementary features is necessary to cope with various changes in appearance. From this point of view, we describe the feature extraction approaches based on multiple principal component analysis and edge distribution. These features are projected on a new intra-person/extra-person similarity space that consists of several simple similarity measures, and are finally evaluated by a support vector machine supervisor. From the experiments on a realistic and large database, an equal error rate of 0.029 is achieved, which is a sufficiently practical level for many real-world applications.

Recently, service area has been emerging field of robotic applications. Even though assistant robots play an important role for the disabled and the elderly, they still &er from operating the robots using conventional interface devices such as joysticks or keyboards. In this paper, we suggest a humanrobot interface using a new eye gaze estimation system. The eye gaze estimation system can estimate the direction of a user’s eye gaze. The proposed system consisted of five IR LEDs and a CCD camera. The direction of the user’s eye gaze can be computed without computing the geometrical relation among the eye, the camera, and the monitor in 30 space. Our method is comparatively simple and fast. The disabled, who is especially people with motor disabilities, can control a robot using this system. We will introduce the method and show experimental results. The experimental results will verify the feasibility of the proposed system as a human-robot interface for the disabled.

A kinematic modeling method is proposed which uses augmenting variables to match the dimensions of the input vector and output vector of each serial subchain of the mobile robots. Firstly, kinematic models of various type of wheels are derived including skidding and sliding velocities. Then the method of augmented generalized coordinates is applied to obtain inverse and forward kinematic models. The kinematic models derived by the direct inverse of the augmented matrices provide an accurate solution, which was not achievable by the modeling method based on the pseudo-inverse method which provides an approximate solution. Lastly, two kinematic models for typical, differential-driven mobile robots are presented to show the effectiveness of the proposed modeling method.