Abstract Dropout is perhaps the most popular regularization method for deep learning. Due to the stochastic nature of the Dropout mechanism, the convergence analysis of Dropout learning is challenging and the existing convergence results are mainly of probability nature. In this paper, we investigate the deterministic convergence of the mini-batch gradient learning method with Dropconnect and penalty. By drawing and presenting a set of samples of the mask matrix of Dropconnect regularization into the learning process in a cyclic manner, we establish an upper bound of the norm of the weight vector sequence and prove that the gradient of the cost function, the cost function itself, and the weight vector sequence deterministically converge to zero, a constant, and a fixed point respectively. Considering Dropout is mathematically a specific realization of Dropconnect, the established theoretical results in this paper are also valid for Dropout learning. Illustrative simulations on the MNIST dataset are provided to verify the theoretical analysis.

Group regularization is commonly employed in network pruning to achieve structured model compression. However, the rationale behind existing studies on group regularization predominantly hinges on the sparsity capabilities of$L_{p}$regularizers. This singular focus may lead to erroneous interpretations. In response to these limitations, this article proposes a novel framework for evaluating the penalization efficacy of group regularization methods by analyzing the impact of$L_{p}$regularizers on weight magnitudes and weight group magnitudes. Within this framework, we demonstrate that$L_{1,2}$regularization, contrary to prevailing literature, indeed exhibits favorable performance in structured pruning tasks. Motivated by this insight, we introduce a hybrid group regularization approach that integrates$L_{1,2}$regularization and group$L_{1/2}$regularization (denoted as HGL1,2&$L_{1/2}$). This novel method addresses the challenge of selecting appropriate$L_{p}$regularizers for penalizing weight groups by leveraging$L_{1,2}$regularization for penalizing groups with magnitudes exceeding a critical threshold while employing group$L_{1/2}$regularization for other groups. Experimental evaluations are conducted to verify the efficiency of the proposed hybrid group regularization method and the viability of the introduced framework.

We propose a novel slot-pattern-control based coded compressed sensing for unsourced random access with an outer A -channel code capable of correcting errors. Specifically, an RM extension code called the patterned Reed-Muller (PRM) code is designed as the inner codeword. We demonstrate the high spectral efficiency due to its enormous sequence space and prove the geometry property in the complex domain so as to propose a projective decoder. Besides, the “patterned” property of the PRM code, which partitions the binary vector space into several subspaces, is further extended as the principle for designing a Slot-Pattern-Control (SPC) pool. By using this pool, the number of simultaneous transmissions in each slot can be vastly reduced. The proposed inner code scheme is then employed in conjunction with two practical outer A -channel codes, and the optimal setups of the system are determined to minimize the required signal-noise ratio (SNR). Our simulation results confirm that the proposed scheme compares favorably with benchmarks regarding the SNR requirement to meet a target error probability.

Widely linear modeling is an important quaternion signal processing technique for capturing the complete second-order statistics of quaternion signals. However, the algorithms based on widely linear modeling are computationally expensive due to the augmented variables and statistics. In this letter, a fast estimation technique based on a real augmented representation of widely linear modeling is proposed in the context of quaternion extreme learning machine with augmented hidden layer (QELMAH), resulting in a reduction of almost 93% of multiplications and 75% of additions. An equivalence between the proposed algorithm and the original QELMAH is theoretically established by proving that the trained networks with the proposed algorithm and the original one mathematically implement identical mapping. Such a technique is also applicable to the widely linear quaternion recursive least squares algorithm. The theoretical analysis and the effectiveness of the proposed algorithms are validated by simulations on two benchmark problems.

In emergency situations such as earthquakes, the cellular infrastructure cannot support communication services because of equipment damage. The use of the large number of unmanned aerial vehicles (UAVs) has been drawn significant attentions as an important solution for providing air-to-ground communication services in such situations. In this paper, we research the flight direction policy (velocity vector) of the UAVs where every UAV acts as the base station to serve the multi-users communications. As the trajectory of UAVs have a huge impact on the performance of communication, we investigate an adaptive coverage problem, that all the UAVs can adjust their velocities to increase the number of served users. However, such behavior may cause larger flight energy consumption. We propose a discrete-time mean-field game (MFG) framework that each UAV adjusts its velocity in order to minimize the flight energy consumption. In this framework, each UAV evolves according to the dynamic equation and seeks to minimize its flight energy consumption containing the average distribution of all UAVs. We investigate a deterministic function φ to approximate the average distribution of all UAVs as the number of UAVs tends to infinity. Furthermore, the optimal velocity vectors generate a certain asymptotic Nash equilibrium as time tends to infinity, which implies that the flight energy consumption of each UAV can reach its minimal value as the number of UAVs increases to infinity. The simulation results show the optimal trajectory and optimal flight tendency of the UAVs. Moreover, we show that as users move, the amount of the users served is maintained at a relatively stable range, which represents met the demand of user's adaptive coverage.

Polar codes can provably achieve the capacity of a symmetric binary discrete memoryless channel and have low encoding and decoding complexity. However, the error rate performance of polar codes decoding in short and moderate length is not very well, moreover, the encoding and decoding of polar codes with the conventional serial mode will lead to poor throughput. In this paper, we propose a hardware architecture of parallel encoding and decoding scheme for polar codes concatenation with LDPC, and take advantage of the parallelism of belief propagation (BP) decoding algorithm of the two codes to reduce the decoding delay. We compare the performance of concatenated scheme with polar codes and investigate the throughput implemented on graphic processing unit (GPU) for Gaussian channel. Experiment results show that the performance of the concatenated scheme outperform only polar codes, and the throughput of the proposed parallel architecture is obviously faster than that of the serial.

In this paper, we study a multi-pair bidirectional (or two-way) full-duplex (FD) massive MIMO relay (FDMMR) system, where the relay employs massive antennas and is operated in the amplify-and-forward (AF) mode. We analyze its spectral efficiency (SE) performance, when both imperfect channel estimation and channel aging effect are considered. We propose four power-scaling schemes based on the zero-forcing reception/zero-forcing transmission (ZFR/ZFT) relaying processing. Assuming that the number of relay antennas approaches infinity, the SE is analyzed in the context of the proposed power scaling schemes. Our analytical results show that the inter-pair interference caused by the other user pairs as well as the self-interference can be completely eliminated by the ZFR/ZFT processing at the relay. The self-loop interference and the inter- user interference can also be cancelled, if a power scaling scheme is carefully selected. Furthermore, our studies show that the channel aging may significantly degrade the SE of the system.

In this paper, a multi-user multi-relay network with wireless energy harvesting (EH) and transferring (ET) is studied. In our system, a simultaneous two-level cooperation, i.e., information-level and energy-level cooperation is conducted for uplink data transmissions (from the users to a destination). Specifically, network coding is employed at the relays to facilitate the information-level cooperation; meanwhile, ET is adopted to share the harvested energy among the users for the energy-level cooperation. The energy minimization problem that takes into account the energy causality and outage probability constraints is formulated. However, the optimization problem is non-convex and hard to be solved directly. Alternatively, an approximation technique is adopted to convert it into a convex one. By solving the convex problem, efficient power allocation and ET policies are designed. Numerical results show that the proposed algorithm is able to achieve a near-optimal performance and outperforms the state of arts.

In this paper, a multi-user multi-relay network with integrated energy harvesting and transferring (IEHT) strategy is studied. In our system, a simultaneous two-level cooperation, i.e., information- and energy-level cooperation is conducted for uplink data transmissions (from the users to a destination). Specifically, network coding is employed at the relays to facilitate the information-level cooperation; meanwhile, ET is adopted to share the harvested energy among the users for the energy-level cooperation. For generality purposes, the Nakagami-m fading channels that are independent but not necessarily identically distributed (i.n.i.d.) are considered. The problem of energy efficiency maximization under constraints of the energy causality and a predefined outage probability threshold is formulated and shown to be non-convex. By exploiting fractional and geometric programming, a convex form-based iterative algorithm is developed to solve the problem efficiently. Close-to-optimal power allocation and energy cooperation policies across consecutive transmissions are found. Moreover, the effects of relay locations, wireless energy transmission efficiency, battery capacity as well as the existence of direct links are investigated. The performance comparison with the current state of solutions demonstrates that the proposed policies can manage the harvested energy more efficiently.

This paper presents a new supervised method for vessel segmentation in retinal images. This method remolds the task of segmentation as a problem of cross-modality data transformation from retinal image to vessel map. A wide and deep neural network with strong induction ability is proposed to model the transformation, and an efficient training strategy is presented. Instead of a single label of the center pixel, the network can output the label map of all pixels for a given image patch. Our approach outperforms reported state-of-the-art methods in terms of sensitivity, specificity and accuracy. The result of cross-training evaluation indicates its robustness to the training set. The approach needs no artificially designed feature and no preprocessing step, reducing the impact of subjective factors. The proposed method has the potential for application in image diagnosis of ophthalmologic diseases, and it may provide a new, general, high-performance computing framework for image segmentation.

Complex gradient methods have been widely used in learning theory, and typically aim to optimize real-valued functions of complex variables. The stepsize of complex gradient learning methods (CGLMs) is a positive number, and little is known about how a complex stepsize would affect the learning process. To this end, we undertake a comprehensive analysis of CGLMs with a complex stepsize, including the search space, convergence properties, and the dynamics near critical points. Furthermore, several adaptive stepsizes are derived by extending the Barzilai-Borwein method to the complex domain, in order to show that the complex stepsize is superior to the corresponding real one in approximating the information in the Hessian. A numerical example is presented to support the analysis.

The scale space-based method has been recently studied for multispectral alignment; however, due to the significant intensity difference between the image pairs, there are usually not enough keypoint correspondences found, and the robustness of the alignment tends to be compromised. In this letter, we attempt to improve the performance from the following two aspects: 1) to avoid the boundary blurring of Gaussian scale space, we adopt nonlinear scale space to explore more keypoints with potential of being correctly matched, and 2) a robust feature descriptor is proposed, and the resulting feature matrix is matched using the previously proposed rotation-invariant distance to obtain more correct keypoint correspondences. Experimental results for multispectral remote images indicate that the proposed method improves the matching performance compared to state-of-the-art methods in terms of correctly matched number of keypoints, aligning accuracy, and rate of correctly matched image pairs. It is also revealed in this letter that, if the descriptor is carefully designed, the local features are distinctive enough for produce good matching even when the main orientation is not present.

Scale invariant feature transform (SIFT) has been widely used in image matching. But when SIFT is introduced in the registration of remote sensing images, the keypoint pairs which are expected to be matched are often assigned two different value of main orientation owing to the significant difference in the image intensity between remote sensing image pairs, and therefore a lot of incorrect matches of keypoints will appear. This paper presents a method using rotation-invariant distance instead of Euclid distance to match the scale invariant feature vectors associated with the keypoints. In the proposed method, the feature vectors are reorganized into feature matrices, and fast Fourier transform (FFT) is introduced to compute the rotation-invariant distance between the matrices. Much more correct matches are obtained by the proposed method since the rotation-invariant distance is independent of the main orientation of the keypoints. Experimental results indicate that the proposed method improves the match performance compared to other state-of-art methods in terms of correct match rate and aligning accuracy.

Normalized cross correlation (NCC) has been widely used to match control points (CP) in image alignment. This method will produce a lot of incorrect matches owing to the significant difference in the image intensity between multispectral image pairs, and furthermore, it is very computationally expensive to handle rotational displacement. This letter presents a method using rotation-invariant distance to match CPs; a local descriptor matrix is built to describe each CP, and fast Fourier transform is introduced to compute the rotation-invariant distance between the matrices. The computational load is sharply decreased by rotation-invariant distance compared to NCC, and furthermore, the load will remain unchanged in circumstance with arbitrary rotational angle. Experimental results indicate that the proposed method improves the match performance compared to other state-of-the-art methods in terms of correct match rate and aligning accuracy.

This letter presents a unified convergence analysis of the split-complex nonlinear gradient descent (SCNGD) learning algorithms for complex-valued recurrent neural networks, covering three classes of SCNGD algorithms: standard SCNGD, normalized SCNGD, and adaptive normalized SCNGD. We prove that if the activation functions are of split-complex type and some conditions are satisfied, the error function is monotonically decreasing during the training iteration process, and the gradients of the error function with respect to the real and imaginary parts of the weights converge to zero. A strong convergence result is also obtained under the assumption that the error function has only a finite number of stationary points. The simulation results are given to support the theoretical analysis.

In this brief, we consider an online gradient method with penalty for training feedforward neural networks. Specifically, the penalty is a term proportional to the norm of the weights. Its roles in the method are to control the magnitude of the weights and to improve the generalization performance of the network. By proving that the weights are automatically bounded in the network training with penalty, we simplify the conditions that are required for convergence of online gradient method in literature. A numerical example is given to support the theoretical analysis.