Although significant progress has been made in the field of speech-driven 3D facial animation recently, the speech-driven animation of an indispensable facial component, eye gaze, has been overlooked by recent research. This is primarily due to the weak correlation between speech and eye gaze, as well as the scarcity of audio-gaze data, making it very challenging to generate 3D eye gaze motion from speech alone. In this paper, we propose a novel data-driven method which can generate diverse 3D eye gaze motions in harmony with the speech. To achieve this, we firstly construct an audio-gaze dataset that contains about 14 hours of audio-mesh sequences featuring high-quality eye gaze motion, head motion and facial motion simultaneously. The motion data is acquired by performing lightweight eye gaze fitting and face reconstruction on videos from existing audio-visual datasets. We then tailor a novel speech-to-motion translation framework in which the head motions and eye gaze motions are jointly generated from speech but are modeled in two separate latent spaces. This design stems from the physiological knowledge that the rotation range of eyeballs is less than that of head. Through mapping the speech embedding into the two latent spaces, the difficulty in modeling the weak correlation between speech and non-verbal motion is thus attenuated. Finally, our TalkingEyes, integrated with a speech-driven 3D facial motion generator, can synthesize eye gaze motion, eye blinks, head motion and facial motion collectively from speech. Qualitative and quantitative evaluations, along with a perceptual user study, demonstrate the superiority of the proposed method in generating diverse and natural 3D eye gaze motions from speech.

The reputation of universities has drawn increasing attention in recent years, especially with the emergence of various rankings. However, despite advances in big data technologies that facilitate data collection and analysis, accurately defining and balancing factors related to university reputation and educational quality remains complex and tedious. Moreover, current educational assessment methods exhibit notable differences and controversies. In this paper, we present Iva , a human-in-the-loop I ntelligent V isual A ssessment system for higher education quality. This system utilizes large language models to analyze extensive multi-modal educational data, with visualization techniques incorporated to enable multi-scale exploration and interaction. Our extensive evaluations, including a carefully-designed user study and expert interviews, demonstrate the system’s potential value and provide insights for future improvements.

With the widespread application and increasing performance of mobile devices, the application demand of 3D models on mobile platforms is constantly expanding. Compared with traditional desktop computers and game consoles, the storage and computing resources of mobile devices are relatively limited, so efficient compression of 3D models is particularly important. To meet this demand, this paper proposes a two-stage pipeline large and small core heterogeneous compression scheme based on CPU, which effectively utilizes the heterogeneous multi-core architecture in mobile devices. In the first stage of the pipeline, the large core is used to compress the mesh topology data, while the super large core is used to compress the vertex data in the second stage. In addition, this paper proposes a parallel optimization of the Edgebreaker algorithm and applies it to the compression of mesh topology structures. This scheme significantly optimizes the allocation of computing resources and compression efficiency. Compared with the traditional single-core CPU compression method, the experimental results show that our method is 24% faster in compression speed on average while maintaining the same compression rate.

Recent advances in AI-driven storytelling have enhanced video generation and story visualization. However, translating dialogue-centric scripts into coherent storyboards remains a significant challenge due to limited script detail, inadequate physical context understanding, and the complexity of integrating cinematic principles. To address these challenges, we propose Dialogue Visualization, a novel task that transforms dialogue scripts into dynamic, multi-view storyboards. We introduce Dialogue Director, a training-free multimodal framework comprising three agents–Script Director, Cinematographer, and Storyboard Maker. This framework leverages large multimodal models and diffusion-based architectures, employing techniques such as Chain-of-Thought reasoning, Retrieval-Augmented Generation, and multi-view synthesis to improve script understanding, physical context comprehension, and cinematic knowledge integration. Experimental results demonstrate that Dialogue Director outperforms state-of-the-art methods in script interpretation, physical world understanding, and cinematic principle application, significantly advancing the quality and controllability of dialogue-based story visualization.

The speech-driven facial animation technology is generally categorized into two main types: 3D and 2D talking face. Both of these have garnered considerable research attention in recent years. However, to our knowledge, the research into 3D talking face has not progressed as deeply as that of 2D talking face, particularly in terms of lip-sync and perceptual mouth movements. The lip-sync necessitates an impeccable synchronization between mouth motion and speech audio. The speech perception derived from the perceptual mouth movements should resemble that of the driving audio. To mind the gap between the two sub-fields, we propose Learn2Talk, a learning framework that enhances 3D talking face network by integrating two key insights from the field of 2D talking face. First, drawing inspiration from the audio-video sync network, we develop a 3D sync-lip expert model for the pursuit of lip-sync between audio and 3D facial motions. Second, we utilize a teacher model, carefully chosen from among 2D talking face methods, to guide the training of the audio-to-3D motions regression network, thereby increasing the accuracy of 3D vertex movements. Extensive experiments demonstrate the superiority of our proposed framework over state-of-the-art methods in terms of lip-sync, vertex accuracy and perceptual movements. Finally, we showcase two applications of our framework: audio-visual speech recognition and speech-driven 3D Gaussian Splatting-based avatar animation.

With the rapid advancement of artificial intelligence, research on enabling computers to assist humans in achieving intelligent augmentation—thereby enhancing the accuracy and efficiency of information perception and processing—has been steadily evolving. Among these developments, innovations in human motion capture technology have been emerging rapidly, leading to an increasing diversity in motion capture data types. This diversity necessitates the establishment of a unified standard for multi-source data to facilitate effective analysis and comparison of their capability to represent human motion. Additionally, motion capture data often suffer from significant noise, acquisition delays, and asynchrony, making their effective processing and visualization a critical challenge. In this paper, we utilized data collected from a prototype of flexible fabric-based motion capture clothing and optical motion capture devices as inputs. Time synchronization and error analysis between the two data types were conducted, individual actions from continuous motion sequences were segmented, and the processed results were presented through a concise and intuitive visualization interface. Finally, we evaluated various system metrics, including the accuracy of time synchronization, data fitting error from fabric resistance to joint angles, precision of motion segmentation, and user feedback.

Parametric 3D models have enabled a wide variety of computer vision and graphics tasks, such as modeling human faces, bodies and hands. In 3D face modeling, 3DMM is the most widely used parametric model, but can’t generate fine geometric details solely from identity and expression inputs. To tackle this limitation, we propose a neural parametric model named DNPM for the facial geometric details, which utilizes deep neural network to extract latent codes from facial displacement maps encoding details and wrinkles. Built upon DNPM, a novel 3DMM named Detailed3DMM is proposed, which augments traditional 3DMMs by including the synthesis of facial details only from the identity and expression inputs. Moreover, we show that DNPM and Detailed3DMM can facilitate two downstream applications: speech-driven detailed 3D facial animation and 3D face reconstruction from a degraded image. Extensive experiments have shown the usefulness of DNPM and Detailed3DMM, and the progressiveness of two proposed applications.

In Linux kernel version 6.6, the Earliest Eligible Virtual Deadline First (EEVDF) scheduler was introduced as the new default scheduler. However, due to its high computational complexity, it may not be suitable for all application scenarios. In cases where there are a large number of short-term tasks or frequent task communication, EEVDF can incur excessive context switch overhead and performance degradation. To address these issues, we propose a lightweight scheduling strategy named SRAND. Leveraging the programmable scheduling framework ‘sched_ext’ and employing BPF technology, we have implemented a strategy based on global and local run queues. This strategy utilizes a five-level BPF mapped queue to partition tasks with different virtual runtimes. Tasks with smaller virtual runtimes are placed into a FIFO-type global queue first, enabling priority scheduling for tasks with smaller virtual runtimes. Additionally, we monitor CPU idle states and allocate tasks in a timely manner, enhancing task responsiveness while reducing scheduling complexity. It is worth noting that our strategy integrates user-space scheduling policies into the kernel via an eBPF program loader, thus eliminating the need for kernel code modifications. By implementing the SRAND strategy, we have observed significant improvements compared to Linux's default scheduling strategy EEVDF. Specifically, our proposed strategy averages an 11.83% reduction in process context switch time and an overall performance improvement of 7.02% in stress tests, while maintaining satisfactory load balancing.

Deep neural networks are being increasingly used for short-term traffic flow prediction, which can be generally categorized as CNNs or GNNs. CNNs typically partition an underlying territory into grid-like spatial units, and employ standard convolutions to learn spatial dependence among the units. However, standard convolutions with fixed geometric structures cannot fully model the nonstationary characteristics of local traffic flows. To overcome the deficiency, we introduce deformable convolution that augments the spatial sampling locations with additional offsets, to enhance the modeling capability of spatial nonstationarity. We design a deep deformable convolutional residual network, namely DeFlow-Net, that can effectively model global spatial dependence, local spatial nonstationarity, and temporal periodicity of traffic flows. Furthermore, to better fit with convolutions, we suggest to first aggregate traffic flows according to pre-conceived regions or self-organized regions based on traffic flows, then dispose to sequentially organized raster images for network input. Extensive experiments on real-world traffic flows demonstrate that DeFlow-Net outperforms GNNs and existing CNNs using standard convolutions, and spatial partition by pre-conceived regions or self-organized regions further enhances the performance. We also demonstrate the advantage of DeFlow-Net in maintaining spatial autocorrelation, and reveal the impacts of partition shapes and scales on deep traffic flow prediction.

Interior scene colorization is vastly demanded in areas such as personalized architecture design. Existing works either require manual efforts to colorize individual objects or conform to fixed color patterns automatically learned from prior knowledge, whilst neglecting user preference. Quantitatively identifying user preferences is challenging, particularly at the early stage of the design process. The 3D setup also presents new challenges as the inhabitant can observe from any possible viewpoint. We propose a representative view selection method based on visual attention and a progressive preference inference model. We particularly focus on the progressive integration of eye-tracked user preference, which enables the assistance in creativity support and allows the possibility of convergent thinking. A series of user studies have been conducted to validate the effectiveness of the proposed view selection method, preference inference model and the creativity support mechanism.

Abstract Land use information is the basis of various geo‐spatial applications. Traditionally, land use patterns are predicted with agent‐based simulation, suffering from a long convergence process. Deep learning techniques have recently been used for land use classification but not prediction, due to the lack and difficulty of collecting enough training data. This paper proposes a novel paradigm for land use data generation with a randomized simulation strategy. We also design a tailored deep land use prediction model, LUPnet, to demonstrate the usage of the paradigm. Experimental results reveal the effectiveness of our method.

Color design for 3D indoor scenes is a challenging problem due to many factors that need to be balanced. Although learning from images is a commonly adopted strategy, this strategy may be more suitable for natural scenes in which objects tend to have relatively fixed colors. For interior scenes consisting mostly of man-made objects, creative yet reasonable color assignments are expected. We propose$C^{3}$C3Assignment, a system providing diverse suggestions for interior color design while satisfying general global and local rules including color compatibility, color mood, contrast, and user preference. We extend these constraints from the image domain to$\mathbb {R}^3$, and formulate 3D interior color design as an optimization problem. The design is accomplished in an omnidirectional manner to ensure a comfortable experience when the inhabitant observes the interior scene from possible positions and directions. We design a surrogate-assisted evolutionary algorithm to efficiently solve the highly nonlinear optimization problem for interactive applications, and investigate the system performance concerning problem complexity, solver convergence, and suggestion diversity. Preliminary user studies have been conducted to validate the rule extension from 2D to 3D and to verify system usability.

Deep learning methods are being increasingly used for urban traffic prediction where spatiotemporal traffic data is aggregated into sequentially organized matrices that are then fed into convolution-based residual neural networks. However, the widely known modifiable areal unit problem within such aggregation processes can lead to perturbations in the network inputs. This issue can significantly destabilize the feature embeddings and the predictions - rendering deep networks much less useful for the experts. This paper approaches this challenge by leveraging unit visualization techniques that enable the investigation of many-to-many relationships between dynamically varied multi-scalar aggregations of urban traffic data and neural network predictions. Through regular exchanges with a domain expert, we design and develop a visual analytics solution that integrates 1) a Bivariate Map equipped with an advanced bivariate colormap to simultaneously depict input traffic and prediction errors across space, 2) a Moran's I Scatterplot that provides local indicators of spatial association analysis, and 3) a Multi-scale Attribution View that arranges non-linear dot plots in a tree layout to promote model analysis and comparison across scales. We evaluate our approach through a series of case studies involving a real-world dataset of Shenzhen taxi trips, and through interviews with domain experts. We observe that geographical scale variations have important impact on prediction performances, and interactive visual exploration of dynamically varying inputs and outputs benefit experts in the development of deep traffic prediction models.

Capturing 3D foot models is important for applications such as manufacturing customized shoes and creating clubfoot orthotics. In this paper, we propose a novel prototype, Sensock, to offer a fully wearable solution for the task of 3D foot reconstruction. The prototype consists of four soft stretchable sensors, made from silk fibroin yarn. We identify four characteristic foot girths based on the existing knowledge of foot anatomy, and measure their lengths with the resistance value of the stretchable sensors. A learning-based model is trained offline and maps the foot girths to the corresponding 3D foot shapes. We compare our method with existing solutions using red-green-blue (RGB) or RGBD (RGB-depth) cameras, and show the advantages of our method in terms of both efficiency and accuracy. In the user experiment, we find that the relative error of Sensock is lower than 0.55%. It performs consistently across different trials and is considered comfortable and suitable for long-term wearing.

Despite numerous progresses in the past decades, 3D shape acquisition techniques remain a threshold for various 3D face based applications. Moreover, advanced 2D data generative models based on the deep networks may not be directly applicable for 3D objects. In this work, we propose a geometry sampling approach to bridge the gap between unstructured 3D face models and the powerful deep networks towards an unsupervised 3D face generative model. Specifically, we devise a geometry sampling approach to obtain a structured representation of 3D faces, which enable us to adapt the 3D faces to the Deep Convolution Generative Adversarial Network (DCGAN) for 3D face generation. We have demonstrated the effectiveness of our generative model by producing a large variety of 3D faces with different facial expressions.

This work explores the use of foot gestures for locomotion in virtual environments. Foot gestures are represented as the distribution of plantar pressure and detected by three sparsely-located sensors on each insole. The Long Short-Term Memory model is chosen as the classifier to recognize the performer's foot gesture based on the captured signals of pressure information. The trained classifier directly takes the noisy and sparse input of sensor data, and handles seven categories of foot gestures (stand, walk forward/backward, run, jump, slide left and right) without manual definition of signal features for classifying these gestures. This classifier is capable of recognizing the foot gestures, even with the existence of large sensor-specific, inter-person and intra-person variations. Results show that an accuracy of ~80% can be achieved across different users with different shoe sizes and ~85% for users with the same shoe size. A novel method, Dual-Check Till Consensus, is proposed to reduce the latency of gesture recognition from 2 seconds to 0.5 seconds and increase the accuracy to over 97%. This method offers a promising solution to achieve lower latency and higher accuracy at a minor cost of computation workload. The characteristics of high accuracy and fast classification of our method could lead to wider applications of using foot patterns for human-computer interaction.

Abstract Several applications in shape modeling and exploration require identification and extraction of a 3D shape part matching a 2D sketch. We present CustomCut, an on‐demand part extraction algorithm. Given a sketched query, CustomCut automatically retrieves partially matching shapes from a database, identifies the region optimally matching the query in each shape, and extracts this region to produce a customized part that can be used in various modeling applications. In contrast to earlier work on sketch‐based retrieval of predefined parts, our approach can extract arbitrary parts from input shapes and does not rely on a prior segmentation into semantic components. The method is based on a novel data structure for fast retrieval of partial matches: the randomized compoundk‐NN graph built on multi‐view shape projections. We also employ a coarse‐to‐fine strategy to progressively refine part boundaries down to the level of individual faces. Experimental results indicate that our approach provides an intuitive and easy means to extract customized parts from a shape database, and significantly expands the design space for the user. We demonstrate several applications of our method to shape design and exploration.

Architectural sketching and massing are used by designers to analyze and explore the design space of buildings. This paper describes a novel multi-touch interface for fast architectural sketching and massing of tall buildings. It incorporates a family of multi-touch gestures, enabling one to quickly sketch the 2D contour of a base floor plan and extrude it to model a building with multi-floor structures. Further, it provides a set of gestures to users: select and edit a range of floors; scale contours of a building; copy, paste, and rotate a building, i.e., create a twisted structure; edit profile curves of a building's profile; and collapse and remove a selected range of floors. The multi-touch system also allows users to apply textures or geometric facades to the building, and to compare different designs side-by-side. To guide the design process, we describe interactions with a domain expert, a practicing architect. The final interface is evaluated by architects and students in an architecture Dept., which demonstrates that the system allows rapid conceptual design and massing of novel multi-story building structures.

With the development of ever-expanding wireless sensor networks (WSNs) that are meant to connect physical worlds with human societies, gathering sensory data at a single point is becoming less and less practical. Unfortunately, the alternative in-network data management schemes may fail to operate in the face of communication voids (or holes) in WSNs (especially 3D WSNs). In response to this challenge, we propose harmonic quorum systems (HQSs) as a lightweight data management system for 2D/3D WSNs. HQSs innovate in exploiting a few scalar fields (constructed using pure localized algorithms) to guide data accesses. This liberates HQSs from depending on any routing mechanisms or location services, hence making HQSs efficient and robust against anomalies in WSN topologies. We implement HQSs in TinyOS, and we perform intensive simulations using TOSSIM to validate the performance of HQSs.

Character pose design is one of the most fundamental processes in computer graphics authoring. Although there are many research efforts in this field, most existing design tools consider only character body structure, rather than its interaction with the environment. This paper presents an intuitive sketching interface that allows the user to interactively place a 3D human character in a sitting position on a chair. Within our framework, the user sketches the target pose as a 2D stick figure and attaches the selected joints to the environment (e.g., the feet on the ground) with a pin tool. As reconstructing the 3D pose from a 2D stick figure is an ill-posed problem due to many possible solutions, the key idea in our paper is to reduce solution space by considering the interaction between the character and environment and adding physics constraints, such as balance and collision. Further, we formulated this reconstruction into a nonlinear optimization problem and solved it via the genetic algorithm (GA) and the quasi-Newton solver. With the GPU implementation, our system is able to generate the physically correct and visually pleasing pose at an interactive speed. The promising experimental results and user study demonstrates the efficacy of our method.

Spatial Sketch is a three-dimensional (3D) sketch application that bridges between physical movement and the fabrication of objects in the real world via cut planar materials. This paper explores the rationale and details behind the development of the Spatial Sketch application, and presents our observations from user testing and a hands-on lamp shade design workshop. Finally we reflect upon the relevance of embodied forms of human computer interaction for use in digital fabrication.

This paper presents a new approach for the mesh composition on models with arbitrary boundary topology. After cutting the needed parts from existing mesh models and putting them into the right pose, an implicit surface is adopted to smoothly interpolate the boundaries of models under composition. An interface is developed to control the shape of the implicit transient surface by using sketches to specify the expected silhouettes. After that, a localized Marching Cubes algorithm is investigated to tessellate the implicit transient surface so that the mesh surface of composed model is generated. Different from existing approaches in which the models under composition are required to have pairwise merging boundaries, the framework developed based on our techniques have the new function to fuse models with arbitrary boundary topology.