Since the global outbreak of COVID-19, numerous interactive and dynamically updated dashboards have been deployed worldwide to disseminate critical information to broad audiences. Unlike traditional researcher-led and lab-based visualization studies, these dashboards represent one of the first large-scale, real-world applications of visualization technology to address a global crisis, offering unique insights for the visualization community into how visual systems are adopted in practice. However, these dashboards are gradually disappearing from the internet as the pandemic subsides. To document these historically significant practices, while responding to the visualization community’s call for practice-driven research, we consolidate and review a dataset of 208 COVID-19 dashboards deployed in real-world settings (”in the wild”). We develop a nested hierarchical taxonomy spanning four levels, 12 named categories, and over 70 exploratory tags. Using this taxonomy, we systematically analyze the dashboards to identify their unique characteristics and design space. We conclude with research challenges and future directions. To support further exploration, we provide an interactive browser (available at https://pandemicdashboards.github.io/ ) that catalogs the reviewed dashboards with their design attributes and archived snapshots.

Generative adversarial network (GAN) is a powerful generative model. However, it suffers from gradient vanishing, divergence mismatching and mode collapse. To overcome these problems, we propose a novel GAN, which consists of one generator G and two discriminators (D1, D2). Focusing on the gradient vanishing, Spectral Normalization (SN) and ResBlock are first adopted in D1 and D2. Then, Scaled Exponential Linear Units (SELU) is adopted at last half layers of D2 to further address the problem. To divergence mismatching, relativistic discriminator is adopted in our GAN to make the loss function minimization in the training of generator equal to the theoretical divergence minimization. Concentrating on the mode collapse, D1 rewards high scores for the samples from the data distribution, while D2 favors the samples from the generator conversely. In addition, the minibatch discrimination is adopted in D1 to further address the problem. Extensive experiments on CIFAR-10/100 and ImageNet datasets demonstrate that our GAN can obtain the highest inception score (IS) and lowest Frechet Inception Distance (FID) compared with other state-of-the-art GANs.

The movements of articulators, such as lips, tongue and teeth, play an important role in increasing the language expression capability by unmasking the information hid in text or speech. Hence, it is necessary to deeply mine and visualize the relationship between text, speech and articulatory movements for understanding language in multi-modality and multi-level. As a case study, given text and audio of President Donald John Trump, this paper synthesizes a high quality 3D animation of him speaking with accurate synchronicity between speech and articulators. First, visual co-articulation is modeled by predicting the mapping from text/speech to articulatory movements. Then, based on a reconstructed 3D head model, physiological characteristics and statistical learning are combined to visualize each phoneme. Finally, the visualization results of consecutive phonemes are fused by visual co-articulation model to generate synchronized articulatory animations. Experiments show that the system can not only produce photo-realistic results in front but also distinguish the visual differences among phonemes from unconstrained views.

We address the issues of 3-D head pose estimation and face modeling from a depth image. Given a depth image, random forests are effective for estimating the location and orientation of a person's head. However, the accuracy of the estimation is not high enough. We propose using corrected regression votes. The corrected votes are obtained by considering the cooperation of all trees, leading to significant improvement of head pose estimation accuracy. Based on the head pose estimator, we present a face modeling system. In our system, the face model is generated by aligning a deformable face model to a depth image using an iterative closest point (ICP) algorithm. The novelty of our approach is that an optimal weight for each vertex is incorporated into the ICP algorithm with point to plane constraints. Experiments show that our system can automatically estimate the head pose and generate a realistic face model from a single depth image. We also provide a detailed evaluation that shows the benefits of our approach.

A real-time 3-D facial animation system produces animation for appearance and internal articulators. For appearance, an anatomical model, including the skeleton, muscle, and skin, is built based on anatomical characteristics and a data-driven model is obtained by learning the mapping between texture and depth. Then, the two models are combined to produce animations with various strengths, since the anatomical model can control the animation strength directly and the data-driven model can capture the nuances of facial motion. For internal articulators, tongue tissue arrangements are obtained from medical data. Then, a nonlinear, quasi-incompressible, isotropic, hyperelastic biomechanical model is applied to describe tongue tissues and an anisotropic biomechanical model is applied to reflect the active and passive mechanical behavior of tongue muscle fibers. The tongue animation is simulated using the finite-element method for realism, while the collisions between the tongue and other articulators are simulated with a mass-spring model for efficiency. Experiments show that the system achieves high perceptual evaluation scores and quantitative improvements are demonstrated in the objective evaluation and user studies, compared with the outputs of other systems.

We propose a speech-driven 3D tongue animation system. Firstly, the input speech is analyzed to obtain the phoneme sequence. Next, articulatory movements are predicted from the phoneme sequence using a hidden Markov model (HMM) based framework. The HMMs are trained beforehand using a corpus of human articulatory movements, which are recorded by three electromagnetic articulograph (EMA) sensors glued on the tongue tip, tongue body, and tongue dorsum of a speaker respectively. Finally, the predicted articulatory movements are used to control the deformations of a 3D tongue model. The tongue model is a triangular mesh with three key vertices. The three vertices are chosen so that their positions are in correspondence with the three EMA sensors mentioned above. Our tongue model can achieve various tongue shapes with volume preservation. Experiments show that the generated tongue animations are realistic and synchronize well with the input speech.

This paper presents a joint head pose and facial landmark regression method with input from depth images for real-time application. Our main contributions are: firstly, a joint optimization method to estimate head pose and facial landmarks, i.e., the pose regression result provides supervised initialization for cascaded facial landmark regression, while the regression result for the facial landmarks can also help to further refine the head pose at each stage. Secondly, we classify the head pose space into 9 sub-spaces, and then use a cascaded random forest with a global shape constraint for training facial landmarks in each specific space. This classification-guided method can effectively handle the problem of large pose changes and occlusion. Lastly, we have built a 3D face database containing 73 subjects, each with 14 expressions in various head poses. Experiments on challenging databases show our method achieves state-of-the-art performance on both head pose estimation and facial landmark regression.

Random forest is a useful tool for face alignment/tracking. The method of regressing local binary features learned from random forest has achieved state-of-the-art performance both in fitting accuracy and speed. Despite the great success of this method, it has certain weaknesses: the number of available local binary features is rather limited and is not optimal for face alignment; the binary features inevitably lead to serious jitter when tracking a video sequence. To address these problems, we propose learning probability features from probabilistic random forest (PRF). The proposed PRF is the same as standard random forest except that it models the probability of a sample belonging to the nodes of a tree. By using the probability features, our method significantly outperforms the state-of-the-art in terms of accuracy. It also achieves about 60 fps for locating a few facial landmarks. In addition, our method shows excellent stability in face tracking.

We present a real-time speech-driven facial animation system. In this system, Gaussian Mixture Models (GMM) are employed to perform the audio-to-visual conversion. The conventional GMM-based method performs the conversion frame by frame using minimum mean square error (MMSE) estimation. The method is reasonably effective. However, discontinuities often appear in the sequences of estimated visual features. To solve this problem, we incorporate previous visual features into the conversion so that the conversion procedure is performed in the manner of a Markov chain. After audio-to-visual conversion, the estimated visual features are transformed to blendshape weights to synthesize facial animation. Experiments show that our system can accurately convert audio features into visual features. The conversion accuracy is comparable to a current state-of-the-art trajectory-based approach. Moreover, our system runs in real time and outputs high quality lip-sync animations.

We address the issue of synthesizing real-time expressive facial animation from videos. Given video footage of a person's face, some existing methods track a few facial landmarks to drive the virtual character. Compared with these methods, ours has the following characteristics. 1) We incorporate global texture into a constrained local model to increase the accuracy of facial tracking. 2) To animate a blenshape face model, facial tracking results as well as facial expression recognition results are used to estimate blendshape weights. Experimental results demonstrate that our method is effective for producing realistic expressive facial animations. Moreover, the method does not require facial markers or complex offline pre-processing, these properties make it very easy to use for ordinary users.

Facial animation is useful in human-machine interaction, computer games and teleconferences. We propose a realtime performance-driven facial animation system for ordinary users. The system enables a user to animate an avatar by performing desired facial motions in front of a video camera. First, a constrained local model based approach is used to track facial features of a performer in the video. To increase the tracking accuracy, we propose an efficient method to build a user-specific local texture model. Next, a 3D blendshape face model is fitted to the tracked feature points. To improve the expressiveness of synthesized animations, facial expression recognition results and pre-recorded animation priors are incorporated into the fitting procedure. Finally, facial animations are created using blendshape interpolation. Experiments show that the synthetic facial motions are realistic and quite similar to the facial actions of the performer. By using an ordinary camera, our system provides the user complete control over the generated facial animations.