We study the problem of learning from positive and unlabeled (PU) data in the federated setting, where each client only labels a little part of their dataset due to the limitation of resources and time. Different from the settings in traditional PU learning where the negative class consists of a single class, the negative samples which cannot be identified by a client in the federated setting may come from multiple classes which are unknown to the client. Therefore, existing PU learning methods can be hardly applied in this situation. To address this problem, we propose a novel framework, namely Federated learning with Positive and Unlabeled data (FedPU), to minimize the expected risk of multiple negative classes by leveraging the labeled data in other clients. We theoretically analyze the generalization bound of the proposed FedPU. Empirical experiments show that the FedPU can achieve much better performance than conventional supervised and semi-supervised federated learning methods.

Neural architecture search tries to shift the manual design of neural network (NN) architectures to algorithmic design. In these cases, the NN architecture itself can be viewed as data and needs to be modeled. A better modeling could help explore novel architectures automatically and open the black box of automated architecture design. To this end, this work proposes a new encoding scheme for neural architectures, the Training-Analogous Graph-based ArchiTecture Encoding Scheme (TA-GATES). TA-GATES encodes an NN architecture in a way that is analogous to its training. Extensive experiments demonstrate that the flexibility and discriminative power of TA-GATES lead to better modeling of NN architectures. We expect our methodology of explicitly modeling the NN training process to benefit broader automated deep learning systems. The code is available at https://github.com/walkerning/aw_nas.

As the computing power of modern hardware is increasing strongly, pre-trained deep learning models (e.g., BERT, GPT-3) learned on large-scale datasets have shown their effectiveness over conventional methods. The big progress is mainly contributed to the representation ability of transformer and its variant architectures. In this paper, we study the low-level computer vision task (e.g., denoising, super-resolution and deraining) and develop a new pre-trained model, namely, image processing transformer (IPT). To maximally excavate the capability of transformer, we present to utilize the well-known ImageNet benchmark for generating a large amount of corrupted image pairs. The IPT model is trained on these images with multi-heads and multi-tails. In addition, the contrastive learning is introduced for well adapting to different image processing tasks. The pre-trained model can therefore efficiently employed on desired task after fine-tuning. With only one pre-trained model, IPT outperforms the current state-of-the-art methods on various low-level benchmarks. Code is available at https://github.com/huawei-noah/Pretrained-IPT and https://gitee.com/mindspore/mindspore/tree/master/model_zoo/research/cv/IPT

Neural network pruning is an essential approach for reducing the computational complexity of deep models so that they can be well deployed on resource-limited devices. Compared with conventional methods, the recently developed dynamic pruning methods determine redundant filters variant to each input instance which achieves higher acceleration. Most of the existing methods discover effective subnetworks for each instance independently and do not utilize the relationship between different inputs. To maximally excavate redundancy in the given network architecture, this paper proposes a new paradigm that dynamically removes redundant filters by embedding the manifold information of all instances into the space of pruned networks (dubbed as ManiDP). We first investigate the recognition complexity and feature similarity between images in the training set. Then, the manifold relationship between instances and the pruned sub-networks will be aligned in the training procedure. The effectiveness of the proposed method is verified on several benchmarks, which shows better performance in terms of both accuracy and computational cost compared to the state-of-the-art methods. For example, our method can reduce 55.3% FLOPs of ResNet-34 with only 0.57% top-1 accuracy degradation on ImageNet. The code will be available at https://github.com/huawei-noah/Pruning/tree/master/ManiDP.

Convolutional network compression methods require training data for achieving acceptable results, but training data is routinely unavailable due to some privacy and transmission limitations. Therefore, recent works focus on learning efficient networks without original training data, i.e., data-free model compression. Wherein, most of existing algorithms are developed for image recognition or segmentation tasks. In this paper, we study the data-free compression approach for single image super-resolution (SISR) task which is widely used in mobile phones and smart cameras. Specifically, we analyze the relationship between the outputs and inputs from the pre-trained network and explore a generator with a series of loss functions for maximally capturing useful information. The generator is then trained for synthesizing training samples which have similar distribution to that of the original data. To further alleviate the training difficulty of the student network using only the synthetic data, we introduce a progressive distillation scheme. Experiments on various datasets and architectures demonstrate that the pro-posed method is able to be utilized for effectively learning portable student networks without the original data, e.g., with 0.16dB PSNR drop on Set5 for ×2 super resolution. Code will be available at https://github.com/huawei-noah/Data-Efficient-Model-Compression.

Transformer models have achieved great progress on computer vision tasks recently. The rapid development of vision transformers is mainly contributed by their high representation ability for extracting informative features from input images. However, the mainstream transformer models are designed with deep architectures, and the feature diversity will be continuously reduced as the depth increases, \ie, feature collapse. In this paper, we theoretically analyze the feature collapse phenomenon and study the relationship between shortcuts and feature diversity in these transformer models. Then, we present an augmented shortcut scheme, which inserts additional paths with learnable parameters in parallel on the original shortcuts. To save the computational costs, we further explore an efficient approach that uses the block-circulant projection to implement augmented shortcuts. Extensive experiments conducted on benchmark datasets demonstrate the effectiveness of the proposed method, which brings about 1% accuracy increase of the state-of-the-art visual transformers without obviously increasing their parameters and FLOPs.

Abstract Wireless sensor networks (WSNs) have been widely investigated in the past decades because of its applicability in various extreme environments. As sensors use battery, most works on WSNs focus on energy efficiency issues (e.g., local energy balancing problems) in statically deployed WSNs. Few works have paid attention to the global energy balancing problem for the scenario that mobile sensor nodes can move freely. In this paper, we propose a new routing protocol called global energy balancing routing protocol (GEBRP) based on an active network framework and node relocation in mobile sensor networks. This protocol achieves global energy efficiency by repairing coverage holes and replacing invalid nodes dynamically. Simulation and experiment results demonstrate that the proposed GEBRP achieves superior performance over the existing scheme. In addition, we analyze the delay performance of GEBRP and study how the delay performance is affected by various system parameters.Copyright © 2013 John Wiley & Sons, Ltd.

ABSTRACT The growing popularity of mobile devices in our daily life demands higher throughput of wireless networks. The new communication standard 802.11n has significantly improved throughput because of the use of advanced technologies such as the multiple‐input multiple‐output communication technique. Because mobile devices are usually battery‐operated, power efficiency is critical; on the other hand, delay performance can be improved by transmitting at high power. To address the conflicting requirement of power saving and small delay, power scheduling is needed. In the past, many approaches to power scheduling have been proposed for real‐time applications, but few of them have considered complicated modes of channel state information(CSI) in multiple‐input multiple‐output. In this paper, we study this and classify the CSI into four types, namely, constant, slow fading, fast fading, and unknown. For known CSI, we propose an optimal algorithm for power scheduling. For unknown CSI, we propose an approximate algorithm based on some heuristics. To improve resource utilization, a stochastic delay‐bound method is proposed for fast‐fading condition. Simulation results demonstrate that the performance achieved by the optimal and heuristic algorithms agrees well with the analysis. Copyright © 2012 John Wiley & Sons, Ltd.

Strict delay performance guarantees are required by many applications in wireless sensor networks. Different from traditional approaches, the data aggregation and the stochastic characteristic need to be considered for the delay analysis in sensor networks. In this paper, the problem that how to calculate the stochastic delay bound under different aggregation schemes is solved with the stochastic network calculus. Meanwhile, to support multifunction in sensor networks, the impact of the heterogeneous services is brought into the analysis of the stochastic delay bound. From some numerical evaluations and the simulation, it is shown that a good tradeoff between performance and implementation can be achieved with a comprehensive aggregation scheme.