Abstract The particular characteristics of deaf‐mutes make them more likely to have mental health problems. Due to their particular way of communication, it is more difficult for them to deal with mental health problems than ordinary people. Nowadays, those psychologists who are good at sign language are in short supply, and remote assistance cannot achieve satisfactory results. Therefore, a library of virtual emotional gestures based on electroencephalogram (EEG) was established and a prototype system for mental health examination of deaf‐mutes was proposed, which help deaf‐mutes identify their psychological problems in time and assist medical staff to examine the psychological problems encountered by deaf‐mutes. In addition, the virtual library of emotional gestures is established with the assistance of the chief physician from a 3A hospital in Henan Province. More importantly, the later experiments demonstrate the applicability of this virtual system.

Abstract In this article, we propose a point cloud synthesis method based on stochastic differential equations. We view the point cloud generation process as smoothly transforming from a known prior distribution toward the high‐likelihood shape by point‐level denoising. We introduce a conditional corrector sampler to improve the quality of point clouds. By leveraging Markov chain Monte Carlo sample, our method can synthesize realistic point clouds. We additionally prove that our approach can be trained in an auto‐encoding fashion and reconstruct the point cloud faithfully. Furthermore, our model can be extended on a downstream application of point cloud completion. Experimental results demonstrate the effectiveness and efficiency of our method.

Abstract As an ancient performing art, shadow puppetry is a treasure of Chinese art. However, with the development of society, shadow puppetry becomes less well‐known among the young generation. To preserve and further spread this traditional culture, the digitization of shadow puppets is playing an increasingly important role in shadow puppetry conservation. Despite this, the spread of shadow puppetry culture in modern times still faces many hindrances. In digitalized shadow puppet art, the virtual scenes in certain degree determine the artistic effect of shadow puppet performance. The commonly used method for digital shadow puppet scene construction is via artificially created models which are then placed in corresponding positions. Obviously, this is a cumbersome and time‐consuming task. Therefore, a semantic‐based scene generation method for digital shadow puppet performance scene is proposed in this paper. According to this method, the key information is extracted from the descriptive text using the Chinese text segmentation technology. Meanwhile, we generate semantic scene graphs and search the corresponding shadow puppet models in the model library to construct the virtual scenes of digital shadow puppet performance. In the evaluation experiment, we invited 30 volunteers (10 female and 20 male) who had been exposed to traditional shadow puppet play in their daily lives. As suggested by the experimental results, the digital shadow puppet performance scene generated in this paper exhibit advantages of convenient use and high availability, which largely enhance the effect of digital shadow puppet performance. It should nonetheless be noted that it's not easy to extract the spatial relations from complicated texts, which inevitably limits the effectiveness of our scene generation method. The research work of this paper aims to promote digital shadow puppet technology and provide insights for the inheritance and conservation of traditional shadow puppet art.

The lack of social activities in the elderly for physical reasons can make them feel lonely and prone to depression. With the spread of COVID-19, it is difficult for the elderly to conduct the few social activities stably, causing the elderly to be more lonely. The metaverse is a virtual world that mirrors reality. It allows the elderly to get rid of the constraints of reality and perform social activities stably and continuously, providing new ideas for alleviating the loneliness of the elderly. Through the analysis of the needs of the elderly, a virtual social center framework for the elderly was proposed in this study. Besides, a prototype system was designed according to the framework. The elderly can socialize in virtual reality with metaverse-related technologies and human-computer interaction tools. Additionally, a test was jointly conducted with the chief physician of the geriatric rehabilitation department of a tertiary hospital. The results demonstrated that the mental state of the elderly who had used the virtual social center was significantly better than that of the elderly who had not used it. Thus, virtual social centers alleviated loneliness and depression in older adults. Virtual social centers can help the elderly relieve loneliness and depression when the global epidemic is normalizing and the population is aging. Hence, they have promotion value

Abstract Point cloud generation aims to synthesize point clouds that do not exist in supervised dataset. Generating a point cloud with certain semantic labels remains an under‐explored problem. This paper proposes a formulation called DiffusionPointLabel, which completes point‐label pair generation based on a DDPM generative model (Denoising Diffusion Probabilistic Model). Specifically, we use a point cloud diffusion generative model and aggregate the intermediate features of the generator. On top of this, we propose Feature Interpreter that transforms intermediate features into semantic labels. Furthermore, we employ an uncertainty measure to filter unqualified point‐label pairs for a better quality of generated point cloud dataset. Coupling these two designs enables us to automatically generate annotated point clouds, especially when supervised point‐labels pairs are scarce. Our method extends the application of point cloud generation models and surpasses state‐of‐the‐art models.

Abstract The emergence of digital opera has enriched the stage performance of Chinese opera and expanded its dissemination means. However, the modern spread of traditional Chinese opera still faces hindrances. Digital opera performances require the generation of virtual scenes of the stages and characters. However, traditional virtual scene generation requires workers to build 3D models using modeling software and incorporate them into the performance scene. This article proposes a semantic‐based generation method for Chinese opera performance scenes. First, we analyze the scene description scripts to understand the elements in Chinese opera virtual scenes. The prior probability is subsequently used to learn the model placement rules in the opera scene model. A digital scene suitable for Chinese opera performance is then generated. The final results show that the method can generate natural and receptive opera digital performance scenes. This article's research ideas and achievements are conducive to the promotion of development of Chinese digital opera technology. They possess substantial significance to the inheritance and development of traditional Chinese opera art.

Abstract For autistic children with slow language function, it is a classic and easy way to understand the development of their cognitive ability through simple painting experiments. Due to the lack of professional evaluators for painting assessment of autistic children, this article research and implement an intelligent assessment system for child autism through recognition of portrait sketch components. A portrait sketch database is constructed with the sample size of 30,400 after data expansion. The data consists of two formats: stroke vector sequence and 2D image. Then we propose a joint model coupled with LSTM and CNN features to automatically segment the portrait sketch components. It can perform better segmentation for exaggerated proportion and incomplete components samples. Finally, according to the evaluation criteria of the painting, we design an assessment model for child autism. The experiments are conducted in cooperation with relevant rehabilitation institutions to verify the effectiveness of the system. The analysis results show that our painting assessment system has a good ability to identify autistic tendencies. It can accurately evaluate children's autistic tendencies through “draw‐a‐man” experiments.

The COVID-19 pandemic has had a severe impact on humans’ lives and and healthcare systems worldwide. How to early, fastly and accurately diagnose infected patients via multimodal learning is now a research focus. The central challenges in this task mainly lie on multi-modal data representation and multi-modal feature fusion. To solve such challenges, we propose a medical knowledge enriched multi-modal sequence to sequence learning model, termed MedSeq2Seq. The key components include two attention mechanisms, viz. intra-modal (Ia) and inter-model (Ie) attentions, and a medical knowledge augmentation mechanism. The former two mechanisms are to learn multi-modal refined representation, while the latter aims to incorporate external medical knowledge into the proposed model. The experimental results show the effectiveness of the proposed MedSeq2Seq framework over state-of-the-art baselines with a significant improvement of 1%-2%.

Abstract With the rapid development of human–computer interaction technologies, the new media generation demands novel learning experiences with natural interaction and immersive experience. Considering that digital storytelling is a powerful pedagogical tool for young children, in this paper, we design an immersive storytelling environment that allows multiple players to use naturally interactive hand gestures to manipulate virtual puppetry for assisting narration. A set of multimodal interaction techniques is presented for a hybrid user interface that integrates existing 3D visualization and interaction devices including head‐mounted displays and depth motion sensor. In this system, the young players could intuitively use hand gestures to manipulate virtual puppets to perform a story and interact with props in a virtual stereoscopic environment. We have conducted a user experiment with four young children for pedagogical evaluation, as well as system acceptability and interactivity evaluation by postgraduate students. The results show that our framework has great potential to stimulate learning abilities of young children through collaboration tasks. The stereoscopic head‐mounted display outperformed the traditional monoscopic display in a comparison between the two.

Digital techniques have been used to assist narrative and storytelling, especially in many pedagogical practices. With the rapid development of HCI techniques, saturated with digital media in their daily lives, young children, demands more interactive learning methods and meaningful immersive learning experiences. In this paper, we propose a novel hand gesture-based puppetry storytelling system which provides a more intuitive and natural human computer interaction method for young children to develop narrative ability in virtual story world. Depth motion sensing and hand gestures control technology is utilized in the implementation of user-friendly interaction. Young players could intuitively use hand gestures to manipulate virtual puppet to perform story and interact with different items in virtual environment to assist narration. Based on the result of the evaluation, this novel digital storytelling system shows positive pedagogical functions on children’s narrating ability as well as the competencies of cognitive and motor coordination. The usability of the system is preliminary examined in our test, and the results which showed that young children can benefit from playing with Puppet Narrator.

Abstract Puppetry has been a popular art form for many centuries in different cultures, which becomes a valuable and fascinating heritage assert. Traditional Chinese marionette art with over 2000 years history is one of the most representative forms offering a mixture of stage performance of singing, dancing, music, poetry, opera, story narrative and action. Apart from a set of string rules, which controls the dynamics, head carving skill is another important pillar in this art form. This paper addresses the heritage preservation of the marionette head carving by digitalizing the head models with a novel modelling technique using ordinary differential equations (ODEs). The technique has been specially tailored to suit the modelling complexity and the need of accurate description of shapes. It offers smoothly sewing ODE swept patches to represent the distinct features of a marionette head with sharp variance of local geometry. Such features otherwise are difficult to model and capture accurately, which may require a great effort and tedious handcrafting of an experienced modeller, when using other representation forms like polygons. Copyright © 2015 John Wiley & Sons, Ltd.