Learning to read Chinese characters poses a considerable challenge for children, particularly those with dyslexia. We designed a tangible system, named Character Alive, to assist children aged 5 to 7 in mastering Chinese radicals and characters. The system employs animations and color cues to clarify the concepts of radicals and morphemes. It also incorporates physical radical cards that work in synergy with an artificial intelligence module to facilitate stroke tracing and character composition. We conducted a two-week within-group study involving 20 six-year-old children who used both our tangible system and an equivalent multitouch version to learn Chinese radicals and characters. The results showed that both systems significantly improved the children’s accuracy in character reading and composition. However, our tangible design outperformed the multitouch variant by increasing children’s learning motivation, enabling hands-on actions, and promoting a diverse range of beneficial learning strategies. We discuss the design implications for creating effective tangible systems to facilitate Chinese language acquisition among children.

• We designed an interactive system that utilizes narrative animations, multilevel games, and affective computing to support children with autism spectrum disorder (ASD) learning neurotypical emotions and facial expressions in social contexts. • Children with ASD achieved generalized learning gains in recognizing and producing neurotypical facial expressions. • The three-step design approach is feasible for developing interactive AI systems for children with ASD. • Narrative animations provide value social contexts, while multilevel games and interactivity help children to stay focused. • Blending AI and human support enables children's personalized, engaging, and effective learning. Emotional skills are crucial for a child's success, though children with autism spectrum disorder (ASD) face challenges in understanding social contexts, as well as recognizing and expressing facial expressions. We present EmoLand, a web-based interactive system that utilizes narrative animations, multilevel games, and artificial intelligence (AI) affective computing to instruct children with ASD about neurotypical emotions and facial expressions in social contexts. The paper sets out the iterative design process used, which was informed by ASD intervention theories, empirical studies focused on children with ASD and their educators (tutors, therapists, parents), and multiple discipline expert inputs. A preliminary evaluation involving twelve children with ASD (aged 4 to 7 years) and five tutors suggests that EmoLand is a viable teaching aid for ASD educators and effectively assists children in learning and applying generalized knowledge about emotions and facial expressions. In this paper, we discuss the design approach and key lessons learned for creating interactive AI systems for children with ASD and explore beneficial considerations for enhancing emotional development.

Integrating physical learning materials with mobile device applications may have benefits for early childhood learning. We present three techniques for creating a hybrid tangible-augmented reality (T-AR) enabling technology platform. This platform enables researchers to develop applications that use readily available physical learning materials, such as letters, numbers, symbols or shapes. The techniques are visual marker-based; computer-vision and machine-learning; and capacitive touches. We describe details of implementation and demonstrate these techniques through a use case of a reading tablet app that uses wooden/plastic letters for input and augmented output. Our comparative analysis revealed that the machine-learning technique most flexibly sensed different physical letter sets but had variable accuracy impacted by lighting and tracking lag at this time. Lastly, we demonstrate how this enabling technology can be generalized to a variety of early learning apps through a second use case with physical numbers.

Realistic recordings of soundscapes often have multiple sound events co-occurring, such as car horns, engine and human voices. Sound event retrieval is a type of contentbased search aiming at finding audio samples, similar to an audio query based on their acoustic or semantic content. State of the art sound event retrieval models have focused on single-label audio recordings, with only one sound event occurring, rather than on multi-label audio recordings (i.e., multiple sound events occur in one recording). To address this latter problem, we propose different Deep Learning architectures with a Siamesestructure and a Pairwise Presence Matrix. The networks are trained and evaluated using the SONYC-UST dataset containing both single- and multi-label soundscape recordings. The performance results show the effectiveness of our proposed model.

Whether literally or suggestively, the concept of soundscape is alluded in both modern and ancient music. In this study, we examine whether we can analyze and compare Western and Chinese classical music based on soundscape models. We addressed this question through a comparative study. Specifically, corpora of Western classical music excerpts (WCMED) and Chinese classical music excerpts (CCMED) were curated and annotated with emotional valence and arousal through a crowdsourcing experiment. We used a sound event detection (SED) and soundscape emotion recognition (SER) models with transfer learning to predict the perceived emotion of WCMED and CCMED. The results show that both SER and SED models could be used to analyze Chinese and Western classical music. The fact that SER and SED work better on Chinese classical music emotion recognition provides evidence that certain similarities exist between Chinese classical music and soundscape recordings, which permits transferability between machine learning models.

Soundscape emotion recognition (SER) aims at the automatic recognition of emotions perceived in soundscape recordings. To benchmark SER, we propose a dataset of audio samples called Emo-Soundscapes and two evaluation protocols for machine learning models. We curated 600 soundscape recordings from Freesound.org and mixed 613 audio clips from a combination of these. The Emo-Soundscapes dataset contains 1213 6-second Creative Commons licensed audio clips. We collected the ground truth annotations of perceived emotion in these 1213 soundscape recordings using a crowdsourcing listening experiment, where 1182 annotators from 74 different countries rank the audio clips according to the perceived valence and arousal. This dataset allows studying SER and how the mixing of various soundscape recordings influences their perceived emotion. The dataset is at http://metacreation.net/emo-soundscapes/.

A music video (MV) is a videotaped performance of a recorded popular song, usually accompanied by dancing and visual images. In this paper, we outline the design of a generative music video system, which automatically generates an audio-video mashup for a given target audio track. The system performs segmentation for the given target song based on beat detection. Next, according to audio similarity analysis and color heuristic selection methods, we obtain generated video segments. Then, these video segments are truncated to match the length of audio segments and are concatenated as the final music video. An evaluation of our system has shown that users are receptive to this novel presentation of music videos and are interested in future developments.