The Bj⊘ntegaard Delta rate (BD-rate) measurements have been used as the primary metrics to evaluate performance of video codecs. However, current BD-rate calculation methods are only applicable under the condition that the rate-distortion (R-D) values maintain a monotonic relationship, as this prerequisite is essential for computing integral along the distortion axis. To address this limitation, we propose a curve-fitting based BD-rate solution that guarantees the reconstructed R-D curve to be monotonic. Considering different use cases, we provide a four parameters logistic curve and a constraint cubic curve to approximate the underlying R-D curve. Computation of BD-rate and BD-metric using fitted R-D curve are elaborated in detail. Experimental results indicate that the proposed solutions work well on non-monotonic data. Furthermore, we verified through quantitative analysis that curve-fitting solutions provide more precise measurements of coding efficiency compared to interpolation methods. This improved accuracy contributed by the proposed methods is attributed to the higher resilience to the inherent randomness present in observed data. The proposed method has been adopted by the MPEG WG4 VCM study group for standardization activities. The source code was released at https://multimedia.tencent.com/resources/tvd.

Medication combination prediction (MCP) can provide assistance for experts in the more thorough comprehension of complex mechanisms behind health and disease. Many recent studies focus on the patient representation from the historical medical records, but neglect the value of the medical knowledge, such as the prior knowledge and the medication knowledge. This article develops a medical-knowledge-based graph neural network (MK-GNN) model which incorporates the representation of patients and the medical knowledge into the neural network. More specifically, the features of patients are extracted from their medical records in different feature subspaces. Then these features are concatenated to obtain the feature representation of patients. The prior knowledge, which is calculated according to the mapping relationship between medications and diagnoses, provides heuristic medication features according to the diagnosis results. Such medication features can help the MK-GNN model learn optimal parameters. Moreover, the medication relationship in prescriptions is formulated as a drug network to integrate the medication knowledge into medication representation vectors. The results reveal the superior performance of the MK-GNN model compared with the state-of-the-art baselines on different evaluation metrics. The case study manifests the application potential of the MK-GNN model.

Sharpening is a widely adopted technique to improve video quality, which can effectively emphasize textures and alleviate blurring. However, increasing the sharpening level comes with a higher video bitrate, resulting in degraded Quality of Service (QoS). Furthermore, the video quality does not necessarily improve with increasing sharpening levels, leading to issues such as over-sharpening. Clearly, it is essential to figure out how to boost video quality with a proper sharpening level while also controlling bandwidth costs effectively. This paper thus proposes a novel Frequency-assisted Sharpening level Prediction model (FreqSP). We first label each video with the sharpening level correlating to the optimal bitrate and quality tradeoff as ground truth. Then taking uncompressed source videos as inputs, the proposed FreqSP leverages intricate CNN features and high-frequency components to estimate the optimal sharpening level. Extensive experiments demonstrate the effectiveness of our method.

Static meshes with texture map are widely used in modern industrial and manufacturing sectors, attracting considerable attention in the mesh compression community due to its huge amount of data. To facilitate the study of static mesh compression algorithm and objective quality metric, we create the Tencent – Static Mesh Dataset (TSMD) containing 42 reference meshes with rich visual characteristics. 210 distorted samples are generated by the lossy compression scheme developed for the Call for Proposals on polygonal static mesh coding, released on June 23 by the Alliance for Open Media Volumetric Visual Media group. Using processed video sequences, a large-scale, crowdsourcing-based, subjective experiment was conducted to collect subjective scores from 74 viewers. The dataset undergoes analysis to validate its sample diversity and Mean Opinion Scores (MOS) accuracy, establishing its heterogeneous nature and reliability. State-of-the-art objective metrics are evaluated on the new dataset. Pearson and Spearman correlations around 0.75 are reported, deviating from results typically observed on less heterogeneous datasets, demonstrating the need for further development of more robust metrics. The TSMD, including meshes, PVSs, bitstreams, and MOS, is made publicly available at the following location: https://multimedia.tencent.com/resources/tsmd.

We propose a multi-task learning neural network for No-Reference image quality assessment (NR-IQA). The pro-posed architecture consists of a backbone feature extractor, a nested multi-task generative module and a quality regression module. We adopt a coarse-to-fine strategy to predict objective error maps in two subtasks optimized with different loss functions. The network is designed to be nested such that discriminative features learned from subtasks are efficiently shared by the primary task. Perceptual distortion maps are achieved by applying masking mechanism between reconstructed error maps and the learned distortion sensitivity map. At last, a quality regression module is adopted to nonlinearly map masked distortions to the subjective score. Experimental results demonstrate the superior performances of the proposed model over state-of-the-art models. The implementation of our method is released at https://github.com/R-JunmingChen/NEMG-IQA.

We propose an end-to-end deep neural network-based approach for full-reference video quality assessment (VQA). Many VQA methods predict local quality first and then apply a pooling mechanism to get a global score. However, these two steps are mostly independent of each other thus could not consider spatial and temporal information of a video simultaneously. The proposed method combines local quality prediction and aggregation together with a unified network that is trained in an end-to-end manner. To be specific, we predict the quality of local spatiotemporal segments with an attention-based convolutional neural network. Furthermore, we propose a spatiotemporal aggregation network (STAN) to allocate adaptive weights to localized quality scores. The aggregation network adopts Convolutional 3D (C3D) kernels to provide a better representation to characterize the quality of videos. Experiment results demonstrate that our method achieves superior performance in comparison with state-of-the-art methods on commonly used video quality datasets.

Unlike traditional 2D image quality assessment, the structural relationship among sub-aperture images (SAIs) is an essential factor affecting the quality evaluation of light field (LF) images, where the labeled datasets are also not sufficient for improving learning performances. To solve these problems, we present a novel deep neural network-based approach to accurately predict the quality of compressed LF images without pristine images. Two modules dubbed SAI-Fusion and Global Context Perception (GCP) are proposed to obtain the relationship among SAIs. For effective training, we compress LF images from EPFL and HCI datasets and propose a ranking-based method to generate pseudo-labels as equivalents of Mean Opinion Score (MOS), i.e., Ranking-MOS. Therefore, we can pre-train our quality assessment network on compressed LF images with Ranking-MOS, and then fine-tune the model at small-scale datasets with real labels. Experiments demonstrate that the proposed method achieves state-of-the-art performance on compressed LF images of Win5-LID dataset.

Network embedding has become a fascinating research subject in recent years owing to its ability to represent networks with rich relationships in the low-dimensional vector space, which inspires various downstream tasks, such as link prediction and node classification. Nevertheless, most existing network embedding methods focus on static networks where nodes and edges do not evolve with time. Although some methods consider the dynamics of networks, they pay little attention to the global information of networks, or have recourse to node labels for training. In this paper, we propose an unsupervised dynamic network embedding using the global information, called UDNGI. More specifically, we first maximize the mutual information between the local node embedding and the global network embedding based on a well-designed graph convolutional network for capturing the global information at a time-step specific snapshot network. Then, a temporal smoothness constraint is proposed to minimize the embedding deviation between two successive snapshots, and a modified long short-term memory is designed to update the weight parameters of the graph convolutional network, which enables the model to capture the global information across all time steps. Extensive experiments on node classification and link prediction demonstrate that UDNGI achieves a generally better performance than state-of-the-art methods.

Point cloud compression is critical to deploy 3D applications like autonomous driving. However, LiDAR point clouds contain many disconnected regions, where redundant bits for unoccupied 3D space and weak correlations between points make it a troublesome problem to achieve efficient compression. This paper aims to aggregate LiDAR point clouds to get compact representations with full consideration of the point distribution characteristics. Specifically, we propose a novel Layer-wise Geometry Aggregation (LGA) framework for LiDAR point cloud lossless geometry compression, which adaptively partitions point clouds into three layers based on the content properties, including a ground layer, an object layer, and a noise layer. The aggregation algorithms are delicately designed for each layer. Firstly, the ground layer is fitted to a Gaussian Mixture Model, which can uniformly represent ground points using much fewer model parameters than adopting the original 3D coordinates. Then, the object layer is tightly packed to reduce the space between objects effectively, and a dense layout for points can benefit compression efficiency. Finally, in the noise layer, the difference between neighbor points is reduced by reordering using Morton Code, and the reduced residuals can help saving bit consumption. Experimental results demonstrate that the proposed LGA significantly outperforms competitive methods without prior knowledge by 12.05~23.37% compression ratio gains. Furthermore, the enhanced LGA with prior knowledge shows consistent performance gains than the latest reference software. Additional results also validate the robustness and stability of our proposed scheme with acceptable time complexity.

Medication combination prediction can be applied to the clinical treatment for critical patients with multi-morbidity. The suitable medication combination can help cure patients and keep the treatment medication safe. However, the complexity and uncertainty of clinical circumstances limit the predictive accuracy of medication combination. Thus, this paper proposes a new medication combination prediction model based on the temporal attention mechanism (TAM) and the simple graph convolution (SGC), named as TAMSGC. More specifically, the TAM can capture the temporal sequence information in the medical records, and the SGC is implemented to acquire the medication knowledge from the complicated medication combination. Experiments in a real dataset show that TAMSGC surpasses the baseline models on the predictive accuracy of medication combination.

Traditional video quality assessment (VQA) methods evaluate localized picture quality and video score is predicted by temporally aggregating frame scores. However, video quality exhibits different characteristics from static image quality due to the existence of temporal masking effects. In this paper, we present a novel architecture, namely C3DVQA, that uses Convolutional Neural Network with 3D kernels (C3D) for full-reference VQA task. C3DVQA combines feature learning and score pooling into one spatiotemporal feature learning process. We use 2D convolutional layers to extract spatial features and 3D convolutional layers to learn spatiotemporal features. We empirically found that 3D convolutional layers are capable to capture temporal masking effects of videos. We evaluated the proposed method on the LIVE and CSIQ datasets. The experimental results demonstrate that the proposed method achieves the state-of-the-art performance.

With explosive increase of internet video services, perceptual modeling for video quality has attracted more attentions to provide high quality-of-experience (QoE) for end-users subject to bandwidth constraints, especially for compressed video quality. In this paper, a novel perceptual model for satisfied-user-ratio (SUR) on compressed video quality is proposed by exploiting compressed video bitrate changes and spatial-temporal statistical characteristics extracted from both uncompressed original video and reference video. In the proposed method, an efficient video feature set is explored and established to model SUR curves against bitrate variations by leveraging the Gaussian Processes Regression (GPR) framework. In particular, the proposed model is based on the recently released large-scale video quality dataset, VideoSet, and takes both spatial and temporal masking effects into consideration. To make it more practical, we further optimize the proposed method from three aspects including feature source simplification, computation complexity reduction and video codec adaption. Based on experimental results on VideoSet, the proposed method can accurately model SUR curves for various video contents and predict their required bitrates at given SUR values. Subjective experiments are conducted to further verify the generalization ability of the proposed SUR model.

Off-line handwritten signature verification has practical significance in financial, administrative and judicial field. Traditional methods rely on handcrafted local and global feature descriptors. Recently, researchers start to apply convolutional neural network (CNN) to learn representative features towards this task. However, most of these methods suffer from overfitting. In this paper we propose a novel two-channel CNN network, namely 2-Channel-2-Logit (2C2L), to address this issue. The input to the network is the concatenation of reference and query signatures. The output of convolutional layers are two logits that measure the similarity between reference and query signatures. We explicitly add dropout layers and a 2-Logit layer to make the network less prone to overfitting issues. The proposed method uses only one reference signature for testing and the training system is independent from the queried user. Experiments on the latest GPDS-Synthetic database demonstrate that the proposed method can significantly improve verification accuracy. We improve the performance by a large margin, from 22.24% in state-of-the-art to 9.95% in terms of equal error rate (EER).

In the field of online handwritten signature verification, it is challenging to verify handwritten signature in a writer-independent scenario. In recent years, many researchers have been applying deep neural network methods to the signature verification task. However, these methods have not outperformed traditional methods, especially when the training samples are limited. In this paper, we propose a novel stroke-based bidirectional RNN architecture. The main idea is to split the signature into multiple patches using strokes. Concatenation of query and reference signature pairs are used as input. The proposed method uses two LSTM RNN networks to extract different features. The first one extracts the features of the strokes and the latter extracts the global features of the whole signatures. The results on the BiosecureID dataset demonstrate that our proposed method can reduce the EER by 33.05%, from 5.6% to 3.75% with fewer features and less training samples. Besides, we find that the proposed stroke based RNN network is 5x faster in training and testing time than Non stroke-based RNN network.

To address the label shortage problem and the fixed-size input constraint of CNN models, we propose a no-reference image sharpness assessment method based on rank learning and effective patch extraction. First, we train a Siamese mobilenet network by learning quality ranks among the synthetically blurred and unsharpen seed images without any human label, which provides effective prior knowledge about the appropriate image sharpness. The extracted single branch finetuned on benchmark datasets is used to predict the subjective rating regarding image sharpness. While the performance of CNN based IQA metrics is compromised due to their fixed-size input constraint, we design a multi-scale gradient guided patch extraction method to increase the quality-sensitive input information and boost the performance. Extensive experimental results on the six public datasets demonstrate that our approach outperforms other state-of-the-art no-reference image sharpness assessment metrics.

A large-scale video quality dataset called the VideoSet has been constructed recently to measure human subjective experience of H.264 coded video in terms of the just-noticeable-difference (JND). It measures the first three JND points of 5-second video of resolution 1080p, 720p, 540p and 360p. Based on the VideoSet, we propose a method to predict the satisfied-user-ratio (SUR) curves using a machine learning framework. First, we partition a video clip into local spatial-temporal segments and evaluate the quality of each segment using the VMAF quality index. Then, we aggregate these local VMAF measures to derive a global one. Finally, the masking effect is incorporated and the support vector regression (SVR) is used to predict the SUR curves, from which the JND points can be derived. Experimental results are given to demonstrate the performance of the proposed SUR prediction method.

The just-noticeable-difference (JND) visual perception property has received much attention in characterizing human subjective viewing experience of compressed video. In this work, we quantity the JND-based video quality assessment model using the satisfied user ratio (SUR) curve, and show that the SUR model can be greatly simplified since the JND points of multiple subjects for the same content in the VideoSet can be well modeled by the normal distribution. Then, we design an SUR prediction method with video quality degradation features and masking features and use them to predict the first, second and the third JND points and their corresponding SUR curves. Finally, we verify the performance of the proposed SUR prediction method with different configurations on the VideoSet. The experimental results demonstrate that the proposed SUR prediction method achieves good performance in various resolutions with the mean absolute error (MAE) of the SUR smaller than 0.05 on average.

Evaluation of coding efficiency is traditionally modeled as a continuous rate-distortion (R-D) function, where the peak signal-to-noise ratio (PSNR) is adopted as the quality measure. Although the PSNR-versus-bitrate curve offers some useful tradeoff information between video quality and coding bit-rates, it does not take human perceptual experience into account. In this work, by following the recent image/video quality assessment framework based on the just-noticeable-difference (JND) notion, we conduct a subjective test for HEVC (High Efficiency Video Codec) video to measure the QP value that lies in the boundary of perceptually lossless and lossy coded bit streams for each human subject. This is also known as the first JND point. It is observed that the statistics of the first JND points of 30 subjects follows the normal distribution for a great majority of test sequences. Finally, a machine-learning approach is proposed to predict the mean of the group-based JND distribution based on extracted video features. It is shown by experimental results that the mean JND point can be predicted accurately.

Nerve injury can cause lower limb paralysis and gait disorder. Currently lower limb rehabilitation exoskeleton robots used in the hospitals need more power to correct abnormal motor patterns of stroke patients’ legs. These gait rehabilitation robots are powered by cumbersome and bulky electric motors, which provides a poor user experience. A newly developed gait rehabilitation exoskeleton robot actuated by low-cost and lightweight pneumatic artificial muscles (PAMs) is presented in this research. A model-free proxy-based sliding mode control (PSMC) strategy and a model-based chattering mitigation robust variable control (CRVC) strategy were developed and first applied in rehabilitation trainings, respectively. As the dynamic response of PAM due to the compressed air is low, an innovative intention identification control strategy was taken in active trainings by the use of the subject’s intention indirectly through the estimation of the interaction force between the subject’s leg and the exoskeleton. The proposed intention identification strategy was verified by treadmill-based gait training experiments.

• A large-scale JND-based coded video quality dataset is presented. • The VideoSet contains 220 5-s sequences in four resolutions coded by H.264/AVC. • The subjective test procedure, JND data cleaning and properties are described. • The significance and implications of the VideoSet are discussed. • This work points out a clear path to data-driven perceptual coding. A new methodology to measure coded image/video quality using the just-noticeable-difference (JND) idea was proposed in Lin et al. (2015). Several small JND-based image/video quality datasets were released by the Media Communications Lab at the University of Southern California in Jin et al. (2016) and Wang et al. (2016) [3]. In this work, we present an effort to build a large-scale JND-based coded video quality dataset. The dataset consists of 220 5-s sequences in four resolutions (i.e., 1920 × 1080 , 1280 × 720 , 960 × 540 and 640 × 360 ). For each of the 880 video clips, we encode it using the H.264/AVC codec with QP = 1 , … , 51 and measure the first three JND points with 30 + subjects. The dataset is called the “VideoSet”, which is an acronym for “Video Subject Evaluation Test (SET)”. This work describes the subjective test procedure, detection and removal of outlying measured data, and the properties of collected JND data. Finally, the significance and implications of the VideoSet to future video coding research and standardization efforts are pointed out. All source/coded video clips as well as measured JND data included in the VideoSet are available to the public in the IEEE DataPort (Wang et al., 2016 [4]).

Based on the notion of just noticeable differences (JND), a stair quality function (SQF) was recently proposed to model human perception on JPEG images. Furthermore, a k-means clustering algorithm was adopted to aggregate JND data collected from multiple subjects to generate a single SQF. In this work, we propose a new method to derive the SQF using the Gaussian Mixture Model (GMM). The newly derived SQF can be interpreted as a way to characterize the mean viewer experience. Furthermore, it has a lower information criterion (BIC) value than the previous one, indicating that it offers a better model. A specific example is given to demonstrate the advantages of the new approach.

A compressed video quality assessment dataset based on the just noticeable difference (JND) model, called MCL-JCV, is recently constructed and released. In this work, we explain its design objectives, selected video content and subject test procedures. Then, we conduct statistical analysis on collected JND data. We compute the difference between every two adjacent JND points and propose an outlier detection algorithm to remove unreliable data. We also show that each JND difference group can be well approximated by a normal distribution so that we can adopt the Gaussian mixture model (GMM) to characterize the distribution of multiple JND points. Finally, it is demonstrated by experimental results that the proposed JND analysis performed in the difference domain, called the D-method, achieves a lower BIC (Bayesian information criteria) value than the previously proposed G-method.

Molecular dynamics (MD) is a computer simulation of physical movements of atoms and molecules, which is a very important research technique for the study of biological and chemical systems at micro-scale. Assisted Model Building with Energy Refinement (AMBER) is one of the most commonly used software for MD. However, the microsecond MD simulation of large-scale atom system requires a lot of computation power. In this paper, we propose mAMBER: an Intel Xeon Phi Many-Integrated Core (MIC) Coprocessors accelerated implementation of explicit solvent all-atom classical molecular dynamics (MD) within the AMBER program package. We mAMBER also includes new parallel algorithm using CPUs and MIC coprocessors on Tianhe-2 supercomputer. With several optimizing techniques including CPU/MIC collaborated parallelization, factorization and asynchronous data transfer framework, we can accelerate the sander program of AMBER (version 12) in 'offload' mode, and achieves a 4.17-fold overall speedup compared with the CPU-only sander program.

Molecular Dynamics(MD) is a computer simulation of physical movements of atoms and molecules in the context of N-body simulation, and is an important part of pharmaceutical industry. GROMACS, which is the most popular software for MD, could not perform satisfactorily with large-scale for the limit of computing resources. In this paper, we proposed a method to accelerate GROMACS with offload mode. In this mode, GROMACS could be arranged efficiently with CPU and the Intel® Xeon PhiTM Many Integrated Core (MIC) coprocessors at the same time, making the full use of Tianhe-2 supercomputer resources. To promote the efficiency of GROMACS, we proposed a series of methods, such as synchronization, data reassemble and array reuse. As we known, we are the first to accelerate GROMACS in offload mode on MIC.