We propose Stacked Deep Sets and Quantile Pooling for learning tasks on set data. We introduce Quantile Pooling, a novel permutation-invariant pooling operation that synergizes max and average pooling. Just like max pooling, quantile pooling emphasizes the most salient features of the data. Like average pooling, it captures the overall distribution and subtle features of the data. Like both, it is lightweight and fast. We demonstrate the effectiveness of our approach in a variety of tasks, showing that quantile pooling can outperform both max and average pooling in each of their respective strengths. We also introduce a variant of deep set networks that is more expressive and universal. While Quantile Pooling balances robustness and sensitivity, Stacked Deep Sets enhances learning with depth.

In the federated learning system, parameter gradients are shared among participants and the central modulator, while the original data never leave their protected source domain. However, the gradient itself might carry enough information for precise inference of the original data. By reporting their parameter gradients to the central server, client datasets are exposed to inference attacks from adversaries. In this paper, we propose a quantitative metric based on mutual information for clients to evaluate the potential risk of information leakage in their gradients. Mutual information has received increasing attention in the machine learning and data mining community over the past few years. However, existing mutual information estimation methods cannot handle high-dimensional variables. In this paper, we propose a novel method to approximate the mutual information between the high-dimensional gradients and batched input data. Experimental results show that the proposed metric reliably reflect the extent of information leakage in federated learning. In addition, using the proposed metric, we investigate the influential factors of risk level. It is proven that, the risk of information leakage is related to the status of the task model, as well as the inherent data distribution.

In federated learning (FL), fair and accurate measurement of the contribution of each federated participant is of great significance. The level of contribution not only provides a rational metric for distributing financial benefits among federated participants, but also helps to discover malicious participants that try to poison the FL framework. Previous methods for contribution measurement were based on enumeration over possible combination of federated participants. Their computation costs increase drastically with the number of participants or feature dimensions, making them inapplicable in practical situations. In this paper an efficient method is proposed to evaluate the contributions of federated participants. This paper focuses on the horizontal FL framework, where client servers calculate parameter gradients over their local data, and upload the gradients to the central server. Before aggregating the client gradients, the central server train a data value estimator of the gradients using reinforcement learning techniques. As shown by experimental results, the proposed method consistently outperforms the conventional leave-one-out method in terms of valuation authenticity as well as time complexity.

Proteins are linear polymers that fold into an incredible variety of three-dimensional structures that enable sophisticated functionality for biology. Predicting protein distance with high precision remains challenging, particularly for small protein families. As deep learning achieves remarkable success in many areas, deep learning also allows scientists to predict proteins' three-dimensional structure. As convolutional neural networks have a powerful ability to learn data features at multiple levels of abstraction, we deploy CNN to predict protein distance. To accelerate the training process, we apply a quantum convolutional neural network(QCNN) to improve the protein structure prediction efficiently. For QCNN, the conventional convolutional layer is transformed to a quantum convolution or quanvolutional layer. Since the protein data has large input, we explore the large dimension of these quantum transformations. And in experiments, we compare the different number of layers in QCNN during the training phase. We found the QCNN is similar to CNN that is the deeper layer can get better performance. The simulations show the proposed method can accelerate the convergence while maintaining the performance.

Federated learning is a distributed deep learning method that enables parallel and distributed learning of data on multiple participants, without the need to centrally store it. In intelligent transportation system, it is impractical to gather the vehicle data from the edge devices due to data privacy concerns or network bandwidth limitation. Hence, combining with federated learning to train vehicle data processing models has become one of the popular solutions. However, such computing paradigm is subject to threats posed by malicious and abnormal nodes that greatly reduces the computing power of the neural network when performing calculations in a distributed manner. In this paper, we use the Vehicle Energy Dataset to simulate distributed vehicle data. Based on these data, we propose an unsupervised anomaly removal and neural network model based on federated learning to solve the problem of outlier data on vehicle equipment and analyze the effect of speed on fuel consumption. The results show that with the proposed anomaly removal strategy, MAE and MSE of the trained network are 29% and 36% lower than those without anomaly removal, respectively.

As a comprehensive and unstructured data source describing 3 dimensional (3D) objects, point cloud data has attracted wide interest from researchers. Neural networks (NNs) provide flexible, end-to-end solutions to point cloud processing tasks. Yet, one of the unavoidable problems in training the NNs is the difficulty in collecting and labeling high-quality point cloud data. Existing point cloud datasets are often collected at different scales and densities. Directly combining highly heterogeneous datasets might violate data protection regulations, but also lead to degradation of model performance. In our experiments, naive horizontal federated learning has been applied to alleviate the data privacy concern. However, when the datasets are not identically distributed, the federated model's performance falls far behind the state-of-the-art of centralized models. This paper investigates the union of highly heterogeneous point cloud datasets for the object classification task. In the proposed framework, the federated model is split into shared and personalized modules. The former improves generalizability by bringing multiple data sources into play, while the latter adapts local distribution and avoids convergence at false optima. Experiments show that the proposed framework FedPCN achieves 35.1% higher accuracy than FedAvg. At local test samples, FedPCN also outperforms the locally trained model by 0.5%.

This paper investigates a novel task of talking face video generation solely from speeches.The speech-to-video generation technique can spark interesting applications in entertainment, customer service, and human-computer-interaction industries.Indeed, the timbre, accent and speed in speeches could contain rich information relevant to speakers' appearance.The challenge mainly lies in disentangling the distinct visual attributes from audio signals.In this article, we propose a light-weight, cross-modal distillation method to extract disentangled emotional and identity information from unlabelled video inputs.The extracted features are then integrated by a generative adversarial network into talking face video clips.With carefully crafted discriminators, the proposed framework achieves realistic generation results.Experiments with observed individuals demonstrated that the proposed framework captures the emotional expressions solely from speeches, and produces spontaneous facial motion in the video output.Compared to the baseline method where speeches are combined with a static image of the speaker, the results of the proposed framework is almost indistinguishable.User studies also show that the proposed method outperforms the existing algorithms in terms of emotion expression in the generated videos.

Unsegmented text recognition is a crucial component in financial document processing systems. Financial text materials, such as receipts, transcripts, identification documents, etc. often involve critical personal information. In many circumstances, these data reside in protected servers of different institutions and must not be transferred beyond the institutional firewall. The emerging technology of Federated Learning (FL) provides a data-secure way of uniting isolated datasets in model training. Using the FL framework, text recognition models can be trained with larger collection of image samples. In previous works, federated text recognition models only deal with single-character images and alpha-numeric corpus. Such models are not competent in industrial applications, especially in Chinese text recognition problems. In this paper, we apply federated learning with a deep convolutional network to perform variable-length text string recognition with a large corpus. In our experiments, we compared two prevalent federated learning frameworks, namely, Tensorflow Federated and PySyft. Results show that federated text recognition models can achieve similar or even higher accuracy than models trained on deep learning framework. On a 5-client distributed dataset, the best character accuracy is achieved by TFF at 49.20%. Extensive experiments are also conducted to evaluate the effect of distributed data storage over the performance of trained models. TFF again achieved a maximum character precision of 54.33% with non-distributed dataset.