We introduce T2V2 (**T**ext to **V**oice and **V**oice to **T**ext), a unified non-autoregressive model capable of performing both automatic speech recognition (ASR) and text-to-speech (TTS) synthesis within the same framework. T2V2 uses a shared Conformer backbone with rotary positional embeddings to efficiently handle these core tasks, with ASR trained using Connectionist Temporal Classification (CTC) loss and TTS using masked language modeling (MLM) loss. The model operates on discrete tokens, where speech tokens are generated by clustering features from a self-supervised learning model. To further enhance performance, we introduce auxiliary tasks: CTC error correction to refine raw ASR outputs using contextual information from speech embeddings, and unconditional speech MLM, enabling classifier free guidance to improve TTS. Our method is self-contained, leveraging intermediate CTC outputs to align text and speech using Monotonic Alignment Search, without relying on external aligners. We perform extensive experimental evaluation to verify the efficacy of the T2V2 framework, achieving state-of-the-art performance on TTS task and competitive performance in discrete ASR.

Speech-driven 3D facial animation aims to generate realistic lip movements and facial expressions for 3D head models from arbitrary audio clips. Although existing diffusion-based methods are capable of producing natural motions, their slow generation speed limits their application potential. In this paper, we introduce a novel autoregressive model that achieves real-time generation of highly synchronized lip movements and realistic head poses and eye blinks by learning a mapping from speech to a multi-scale motion codebook. Furthermore, our model can adapt to unseen speaking styles, enabling the creation of 3D talking avatars with unique personal styles beyond the identities seen during training. Extensive evaluations and user studies demonstrate that our method outperforms existing approaches in lip synchronization accuracy and perceived quality. Demos and codes are available at https://xg-chu.site/project_artalk/.

Music visualization is presented in different formats. The mainstream music visualization remains focused on providing animated images for visual pleasure or for enhancing the perception of sheet music with colors. Static visualization of music, on the other hand, has the potential to provide quick insights into the music’s attributes, particularly its mood. However, despite our previous work exploring several approaches to music mood visualization, a major problem still exists in how to objectively evaluate the mood of music pieces and the generated images. Thus, this paper proposes IMEMNextan improvement to the existing framework IMEMNet [1], which serves as a novel and objective metric for music mood visualization. Not only does it allow for a user study-free evaluation of music mood visualization, but the new network also enables for creation of datasets of music-image pairs with the same mood. The new IMEMNext network achieved a significant performance improvement compared to its original implementation.

Music has been visualized in different forms. Majority of the existing methods of music visualization utilized only a select few parameters of the music, of which pitch and frequency were the most visualized. Also, current musk visualizers usually produce animated visual backgrounds for the music being played. Visualization of music as static images was only addressed by some people who claimed to have a neurological condition called synesthesia or chromesthesia. In this paper, we use artificial intelligence to simulate such music visualization. Specifically, we consider how the mood of music can be visualized in static images using deep learning techniques. We consider two approaches. First, we use the deep learning network to generate abstract paintings based on the music sentiments obtained from the Spotify library. In the second approach, two different deep learning networks are used to both classify the musk and to generate the respective landscape images representing its mood. The results are analyzed and compared as well as subjected to user tests.

We performed a feasibility study on new ways of using sound in interactive computer graphics to improve visual interaction as well as to replace or augment haptic interaction. We considered using sound during surface interaction tasks performed with a desktop haptic device and an optical hand tracking device and evaluated the efficiency of such interactive geometry sonification in terms of its precision and speed. We also considered scenarios of using sound for easing navigation while moving along a path or surface.

User interaction is one of the most important features of online social networks, and is the basis of research of user behavior analysis, information spreading model, etc. However, existing approaches focus on the interactions between adjacent nodes, which do not fully take the interactions and relationship between local region users into consideration as well as the details of interaction process. In this paper, we find that there exists influence transfer effect in the process of user interactions, and present a regional user interaction model to analyze and understand interactions between users in a local region by influence transfer effect. Based on real data from Sina Weibo, we validate the effectiveness of our model by the experiments of user type classification, influential user identification and zombie user identification in online social networks. The experimental results show that our model present better performance than the PageRank based method and machine learning method.