We present a novel system that automatically generates and visualizes 3DCG dance animations based on the user’s preferred music and dance composition. The key technology of the system is a transformer-based diffusion model that produces dance choreographies conditioned on arbitrary inputs of music audio and dance composition. Integrated into a user-friendly GUI, the system allows users to instantly generate and preview multiple dance sequences simply by selecting their desired music and dance composition. This capability supports both creative choreography ideation and effective dance practice.

Significant research attention has recently been focused on the automatic generation of human dance choreography from music. While several generation models have been proposed, they cannot specify what kind of movements to generate, and as a result, random movements are generated. We therefore propose a generation model called Composition-specified Dance Choreography Generation from Music (CDCGM) that enables creators to specify which dance composition (i.e., type of movement) to take when generating a dance at each time step. We implemented CDCGM by first constructing a new dataset that includes motion captures of breakdancing and time-series annotation data of representative movement types. Evaluation experiments using our corpus showed that CDCGM can generate dances that faithfully reflect the specified dance composition with high quality. Compared to conventional state-of-the-art models, CDCGM is capable of generating quality dances that improve the expressiveness and the degree to which the dance matches the content and timing of the music. We also propose a new application for CDCGM in which users watch newly generated dance choreography simply by entering music and dance composition. The results of a user study evaluation of the application demonstrated that users found the experience of generating dance by specifying any dance composition for any music extremely fun, that it has the potential to greatly contribute to dance choreography and learning, and that there is a strong desire to use this application on a daily basis.

In this study, we examine how incorporating personality traits into a nonverbal behavior generation model for upper-body motion (head, arms, and posture) affects the quality and characteristics of the generated behaviors. We first constructed a multimodal dialogue corpus containing speech audio, transcripts, 3D upper-body skeleton data, and participants’ Big Five personality scores, and then used the corpus to develop a model that predicts 3D skeleton coordinates from speech, text, and personality traits. Objective evaluation showed that the model with personality input more accurately reproduced individualized behaviors aligned with personality traits. The generated gestures also reflected the relationship between gesture expressivity and personality. Subjective evaluation further showed that observers could reliably perceive intended differences in personality levels—specifically, high vs. low Big Five scores—based only on the generated movements. These findings demonstrate that modeling personality traits enables the generation of agent behaviors that are both personality-consistent and perceptible to users.

We investigated how far into the future nonverbal behavior can be predicted from speech and speech text. Specifically, we build a model that generates future behaviors from speech and speech text information and evaluate the quality of the generated behaviors. This helps to clarify how far into the future behavior can be accurately predicted. Our experimental results show that in Gesture Pose Generation using speech and speech text, on the basis of the input speech and text, the nonverbal behavior up to at least 500 ms ahead can be predicted with objective evaluation values that are the same as those when no future prediction is made. This result shows a new possibility for Gesture Pose Generation using speech and speech text to predict the future up to at least 500 ms ahead with no performance degradation.

In recent years, the human Digital Twin has been discussed as new technology. When we discuss a world in which one’s self-avatar autonomously performs social activities in cyberspace, the questions arise whether or not the behavior of the avatars feels like one’s own, and whether or not we can approve of the self-avatars’ social activities on behalf of ourselves. We define such feeling as the sense of self-individuality. In this study, we focused on the situation in which self-avatars perform presentations on behalf of ourselves to investigate the effect of the modification experience on the presentation motions by self-avatars on the sense of self-individuality. We conducted VR-based experiments in which the motion modification intervention was performed on self-avatars over eight weeks by 24 experiment participants. As a result, we found that the sense of self-individuality was improved as the number of modifications and interventions increased. However, we found that the intensity of motion modification did not correlate with the improvement of the sense of self-individuality in this experiment condition. We also found that the sense of self-individuality was reduced when others intervened in the motion. From these results, we clarified that the experience of motion modification on self-avatars is significant when designing the behavior of avatars acting on behalf of ourselves in human Digital Twin. Further investigation is required to clarify the effect of the long-term intervention on behavior to distinguish between the mere exposure effect.

Demands for agents that replicate the characteristics of specific individuals are increasing. Although ways to implement personality traits into the virtual agents’ movement have been widely researched, ways to create aspects of individuality that can be identified as belonging to specific individuals have not. To clarify how well humans can identify individuals from short movements and what elements of movement contribute to the perception of individuality, we examined the relationship between the degree of confidence in personal identification and statics of gesture movement. In the experiment, participants were asked to compare pairs of short presentation animations and give their degree of confidence that the two animations were of the same person. The animations were created with motion data from performers and with 3D-CG characters to reduce the differences in appearances and shot angles. We calculated five expressivity parameters from the wrist movement for each gesture and compared the answers from the participants. The results showed that the participants were able to distinguish individuals doing the same action and recognize the individuals by the spatial and temporal extents of their movement, which were represented by how much space they use and how fast they moved their wrists. This study clarifies the cognitive aspects of what elements need to be reproduced to develop agents with individuality.

This paper proposes a camera-based geo-fencing system for wireless local area networks (WLANs) which enables geo-location based wireless access control to intuitively manage the area where the WLANs are available. The proposed system leverages camera to localize WLAN users accurately and estimates the proximity of users to objects in the real world. Meanwhile, conventional geo-location based access control suffers from low accuracy of RSSI based localization. As an example of geo-location based access control, we execute a WLAN activation control which allows STAs to pre-activate WLAN and associates with access points (APs) so that the power consumption and time to obtain contents are reduced. Experimental results show the feasibility of camera-based geo-fencing.

Controllable particle display has been proposed, that controls the position and blinking patterns to yield a visual representation and the representation to touch these particles. Additionally, as an example of its implementation, a controllable particle display using water drops as the particles was proposed. In this system, objects are represented by projecting images upward onto falling water drops designed to form a plane surface, depending on the positions of the water drops. However, this method has a problem in terms of the brightness of the object. In this paper, we propose a method by which images are projected onto falling water drops at an angle, and users observe the images from in front of a projector.