Recent advances in research have demonstrated that Vision-Language Models (VLMs) are a promising technology for robot task planning. This paper presents a novel approach that leverages visual prompts and VLMs to generate feasible robot action sequences for achieving shared tasks through human-robot collaboration while simultaneously estimating human intentions. Our method enhances VLMs’ understanding of the environment by utilizing annotations (bounding boxes and labels) and dynamically infers human intentions based on changing environmental conditions to generate optimal robot action sequences to achieve common goals. Additionally, the system incorporates a mechanism to regenerate new sequences through VLM analysis when action failures or external interference occur. Furthermore, by designing prompts as versatile modules for diverse tasks, our proposed technology offers a new approach to robot action planning that excels in both efficiency and adaptability.

This work proposes a cyclic generative adversarial network with spatial-wise and channel-wise attention modules for text-to-face synthesis and manipulation. Then, we explore the pre-trained transformer-based BERT model to obtain text embedding. Furthermore, dual-layer perceptual loss and SSIM loss are introduced to reinforce the delicate features and preserve facial identity during the manipulation task. Additionally, we adopt a novel Flickr-Faces-HQ with Text descriptions (FFHQ-Text) dataset with numerous facial attribute annotations to advance the development of the text-to-face task. In particular, by introducing the StyleGAN encoder for learning latent representations to our proposed post-processing method, we demonstrate that even training on a smaller text-to-face dataset can synthesize more realistic images. Experimental results reveal the effectiveness of our approach, which generates photo-realistic facial images, edits the specific facial attribute with the correlated keywords manipulation, outperforms previous state-of-the-art methods both in quality and quantity, and suggests promising future directions.

In recent studies, the widespread of deep learning has made many kinds of large-scale image datasets available and it has enabled to improve the performance of image-based 3-D scene reconstruction. Several studies estimate whole 3-D scenes including occluded or unseen parts consistent with the obtained partial observations by integrating prior knowledge from training datasets with them, under no camera parameters nor image landmark correspondence are known. Although they generate “discrete” scene instances, they cannot represent and treat their “ambiguity” at all.This paper proposes a novel deep-learning-based framework that can directly represent and treat the ambiguity of scene reconstructions. We introduce a neural network which encodes a target scene as a descriptor. The network takes partial observations as input and outputs a parametric set of the scene descriptors containing all scenes consistent with given observations. The input observations may be “incomplete” in the sense that they do not have enough pieces of information to uniquely determine the whole scene due to neither geometry in-formation nor landmark correspondences available (ill-defined cases). The network is trained based on the dataset of the complete 3-D scenes and possible partial observations so that it can predict the unseen parts from incomplete observations. The paper introduces the method to induce such a descriptor space into the encoder/decoder architecture by employing novel definitions of loss functions measuring “validity”, “consistency” and “reproducibility”. When the series of partial and incom-plete observations for the same 3-D scene is obtained, the reconstruction ambiguity is explicitly treated by parametrically integrating the descriptor set for each observation into one descriptor set.

This research used a questionnaire survey to examine the positive and negative opinions of Japanese university students about living with robots. The results show that the effect of educational background on the hope of living with a robot is statistically significant, that gender affects negative attitudes toward the social influence of robots, and that negative correlation between the hope of living with a robot and negative attitudes toward emotional interaction with robots is statistically significant. An exploratory qualitative classification reveals that most Japanese undergraduates hold the negative opinion that they have no need to live with robots because they are not alone.

Robots are required to support people's work. In order to alleviate the burden on people, it is desirable that robot can automatically generate and execute complicated motions according to simple directions from people. However, there are multiple grasping methods for one object. In order to select a motion suitable for the direction, it is important to estimate candidates of grasping method for the object. In this research, we purpose to recall candidates of grasping position and hand shape from an object image. In learning, a network that outputs a plurality of grasping method candidates for one object image to each channel of a multi-channel image is used. At this time, a plurality of grasping methods are not learned at same time, learned one by one. The similar grasping method for the similar object shape is automatically clustered to each output channel in the learning process, and a grasping method having a characteristic difference is presented as a candidate. We show the usefulness of this method using experimental examples.

The auto-encoder method is a type of dimensionality reduction method. A mapping from a vector to a descriptor that represents essential information can be automatically generated from a set of vectors without any supervising information. However, an image and its spatially shifted version are encoded into different descriptors by an existing ordinary auto-encoder because each descriptor includes a spatial subpattern and its position. To generate a descriptor representing a spatial subpattern in an image, we need to normalize its spatial position in the images prior to training an ordinary auto-encoder; however, such a normalization is generally difficult for images without obvious standard positions. We propose a transform invariant auto-encoder and an inference model of transform parameters. By the proposed method, we can separate an input into a transform invariant descriptor and transform parameters. The proposed method can be applied to various auto-encoders without requiring any special modules or labeled training samples. By applying it to shift transforms, we can achieve a shift invariant auto-encoder that can extract a typical spatial subpattern independent of its relative position in a window. In addition, we can achieve a model that can infer shift parameters required to restore the input from the typical subpattern. As an example of the proposed method, we demonstrate that a descriptor generated by a shift invariant auto-encoder can represent a typical spatial subpattern. In addition, we demonstrate the imitation of a human hand by a robot hand as an example of a regression based on spatial subpatterns.

In this paper, we describe new way to use augmented reality (AR) for entertainment. AR is a technology that overlays virtual objects on real objects in real world as a natural and intuitive interface. We think AR also can provide multiple users with different sight in real world and control information displayed for each user. We use these AR features for entertainment to present another enjoyment for users. As an example of our proposal, we implemented a card game system and evaluated it to examine usefulness of our concept for entertainment.

This paper proposes a system for an automatic detection of indoor scene events with interactive inquiry based on speech dialog and gesture recognition. The system detects the events that various objects are brought in or taken out by image recognition. The user of the system inquires the stored events in the past by pointing the objects or space and using speech dialog. Since automatic event detection may fail in complicated indoor scene, the system can use interactive inquiry to correct such failures.

Sign language is used for communicating to people with hearing difficulties. Recognition of a sign language image sequence is challenging because of the variety of hand shapes and hand motions. We propose a method to automatically construct a transitional structure(topology) of a Hidden Markov Model(HMM) for recognizing sign language words. Unlike conventional HMM, the constructed topology has branches and junctions in order to represent a flexible structure. The proposed method consists of segmentation of a motion, and construction of the topology from segments. The topology is constructed from an initial topology by modifying it. With experiments, we show the effectiveness of the proposed method.

This paper describes a method of probabilistic obstacle map building based on Bayesian estimation. Most active or passive obstacle sensors observe only the most frontal objects and any objects behind them are occluded. Since the observation of distant places includes large depth errors, a conventional method, which does not consider the sensor occlusion often, generate erroneous maps. We introduce a probabilistic observation model, which determines the visible objects. We first estimate probabilistic visibility from the current viewpoint by a Markov chain model based on the knowledge of the average sizes of obstacles and free areas. Then the likelihood of the observations based on the probabilistic visibility are estimated and then the posterior probability of each map grid are updated by Bayesian update rule. Experimental results show that more precise map building can be built by this method.

This paper describes a choice strategy to ease user's burdens for an interactive object recognition system when the system obtains multiple object candidates as a recognition result. First, we propose several methods to display the recognition result so as to make recognition of the candidate easy and hierarchical methods to reduce candidates per choice. We verify their effectiveness by subjective tests. Next, we propose a strategy to minimize time spent for choices. We divide the time into that for displayed candidates and that for speech dialog, and formulate each time to evaluate the strategy quantitatively. Last, we compare the strategy based on choice time with a subjective strategy

With the increase in the number of senior citizens, there is a growing demand for human-friendly wheelchairs as mobility aids. One of the main issues in the robotic wheelchair research is autonomous obstacle avoidance for safety. However, this is difficult because most moving obstacles in the real world are human beings. They sometimes change their motion abruptly. The paper presents an intelligent wheelchair that can avoid collisions with such human pedestrians safely and comfortably for each other. We assume that the information whether or not a pedestrian has noticed the wheelchair and which direction he/she wants to go can appear in the face direction. Thus our intelligent wheelchair is continuously observing the pedestrian's face in its front area, realizing smooth passing by changing its collision avoidance strategy based on the face information and the range data from the ultrasonic sensors. Experimental results show the effectiveness and comfortableness of the proposed method.

Proposes a probabilistic method for detecting people in a corridor using a sequence of color images. In a learning phase, the templates of people and the color properties of skins, a background without shadows, and a background with shadows of people are obtained. In a detection phase, the probabilities of the color of each pixel belonging to various objects are computed. Then templates of a person are placed at various image positions and the criterion of matching to a person is computed for each position. If a template is matched to a person, a more precise position of the person is determined near the matched position. We carry out an experiment using real image sequences to show the effectiveness of the method.

We presented an intelligent wheelchair whose motion can be controlled by the user's face direction. We propose to add intelligence to our wheelchair when the user is not riding. It can recognize the user's face and can move according to the gestures made by the user. Gesture is a good means to give commands because it can be used in noisy conditions and when we would not like to speak loudly in public places. However, environments where wheelchairs are used cannot be controlled. This makes gestures recognition difficult. We propose an interactive way to solve this problem. When the wheelchair is not certain about the meaning of the user's gesture, it guesses the meaning and moves a little accordingly to show its guess to the user. Then it observes the user's response, judging whether its guess is correct or not. This guess-action-observation cycle is repeated until the wheelchair can understand the user's intention.

This paper proposes a method for the rapid and precise estimation of human hand postures by combining 2D appearance matching and 3D model-based fitting. First a rough posture estimate is obtained by image indexing. Each possible hand appearance generated from a given 3D shape model is labeled by an index obtained by PCA compression and registered with its 3D model parameters in advance. By retrieving the index of the input image, the method can obtain the matched appearance image and its 3D parameters rapidly. Then, starting from the obtained rough estimate, it estimates the posture and moreover refines the given initial 3D model by model-fitting. The sequential activation of the two processes in every frame gives the precise posture estimate rapidly. The effectiveness of the method is shown by experimental results.

Humans can communicate with each other by gestures. This paper presents a robot system with such capability. The robot detects its user by recognizing his/her face. It can accept his/her commands given by gestures. The user may use unknown gestures to the robot. If the robot does not respond to his/her gesture, the user usually iterates the same gesture. The robot detects this repetitive pattern as an intentional gesture by which the user wants to give it some order. Then it shows a little action according to the meaning of the gesture that the robot guesses. It observes the user's reaction to its action. If he/she continues the same pattern gesture, the robot considers that its understanding is right, completing the action. It also registers the pattern as a gesture of the guessed meaning. Otherwise, it iterates the same procedure by taking another action as a candidate of the meaning. We have implemented such interactive capability on our intelligent wheelchair.

With the increase in the number of senior citizens, there is a growing demand for human-friendly wheelchairs as mobility aids. One of the main issues in robotic wheelchair research is autonomous obstacle avoidance for safety. However, this is difficult because most of the moving obstacles in the real world are human beings, and they sometimes change their motion abruptly. This paper presents an intelligent wheelchair that can avoid collisions with such pedestrians (and each other) safely and comfortably. We assume that the information as to whether or not a pedestrian has noticed the wheelchair and which direction he/she wants to go can appear in the direction of the pedestrian's face. Thus, our intelligent wheelchair continuously observes the pedestrian's face in its frontal area, realizing smooth passing by changing its collision-avoidance strategy based on the face information and the range data from ultrasonic sensors. The effectiveness of the proposed method is shown by a comparative experiment with a method using only ultrasonic sensors.

With the increase of senior citizens, there is a growing demand for human-friendly wheelchairs as mobility aids. The paper proposes the concept of a robotic wheelchair to meet this need. It has three motion control modes. One is an autonomous control mode. The wheelchair has autonomous navigation capabilities, such as avoiding obstacles and moving straight toward a target object or along a corridor using ultrasonic and vision sensors. Another is a manual control mode with minimum intentional operations. The wheelchair observes the user's face by vision, moving in the direction indicated by the use's face movements. The third is a remote control mode. When the user is not riding, it can find him/her using face recognition. The user can make it come or go by hand gestures. We have developed an experimental robotic wheelchair based on the above concept. Experimental results prove our approach promising.

With the increase in the number of senior citizens, there is a growing demand for human-friendly wheelchairs as mobility aids. To meet this need, we proposed a robotic wheelchair which can be controlled by turning our face in the direction where we would like to go. Although it can be used easily, there is a problem that unintentional movements of our face may interfere with the wheelchair motion. The paper presents our new version of the wheelchair improved by observing both user and environment. It effectively integrates autonomous capabilities and the interface by face direction. It uses the sensor information obtained for autonomous navigation to solve the problem with the control by face direction. Also, if it can understand the user's intentions from observing the face, it chooses an appropriate autonomous navigation function to reduce the user's burden of operation.

With the increase of senior citizens, there is a growing demand for human-friendly wheelchairs as mobility aids. The paper proposes a concept of an intelligent wheelchair to meet this need. It can understand human intentions by observing the user's nonverbal behaviors and can move in accordance with the user's wish with minimum human operations. The paper also describes our experimental robotic wheelchair system. Human intentions appear most on the face. Thus, the experimental system observes the human face, computing its direction. As the first step toward the intelligent wheelchair, we have made experiments on controlling the system's motion by the face direction. Experimental results prove our approach promising.