Due to the extraordinary abilities in extracting complex patterns, graph neural networks (GNNs) have demonstrated strong performances and received increasing attention in recent years. Despite their prominent achievements, recent GNNs do not pay enough attention to discriminate nodes when determining the information sources. Some of them select information sources from all or part of neighbors without distinction, and others merely distinguish nodes according to either graph structures or node features. To solve this problem, we propose the concept of the Influence Set and design a novel general GNN framework called the graph influence network (GINN), which discriminates neighbors by evaluating their influences on targets. In GINN, both topological structures and node features of the graph are utilized to find the most influential nodes. More specifically, given a target node, we first construct its influence set from the corresponding neighbors based on the local graph structure. To this aim, the pairwise influence comparison relations are extracted from the paths and a HodgeRank-based algorithm with analytical expression is devised to estimate the neighbors' structure influences. Then, after determining the influence set, the feature influences of nodes in the set are measured by the attention mechanism, and some task-irrelevant ones are further dislodged. Finally, only neighbor nodes that have high accessibility in structure and strong task relevance in features are chosen as the information sources. Extensive experiments on several datasets demonstrate that our model achieves state-of-the-art performances over several baselines and prove the effectiveness of discriminating neighbors in graph representation learning.

Recent interests in graph neural networks (GNNs) have received increasing concerns due to their superior ability in the network embedding field. The GNNs typically follow a message passing scheme and represent nodes by aggregating features from neighbors. However, the current aggregation methods assume that the network structure is static and define the local receptive fields under visible connections, which consequently fails to consider latent or high-order structures. Besides, the aggregation methods are known to have a depth dilemma due to the over-smoothness issues. To solve the above shortcomings, we present in this article a compact graph convolutional network framework which defines the graph receptive fields based on diffusion paths and explicitly compresses the neural networks with sparsity regularization. The proposed model seeks to learn from invisible connections and recover the latent proximity. First, we infer the high-order proximity and construct diffusion paths by diffusion samplings. Compared with random walk samplings, the diffusion samplings are based on regions instead of paths. The network inference then obtains accurate weights that can be leveraged to build small but informative receptive fields with salient neighbors. Second, to utilize the deep information while avoiding overfitting, we propose learning a lightweight model by introducing a nonconvex regularizer. Numerical comparisons with the existing network embedding methods under unsupervised feature learning and supervised classification show the effectiveness of our model.

Knowledge graph embedding is an effective way to represent knowledge graph, which greatly enhance the performances on knowledge graph completion tasks, e.g., entity or relation prediction. For knowledge graph embedding models, designing a powerful loss framework is crucial to the discrimination between correct and incorrect triplets. Margin-based ranking loss is a commonly used negative sampling framework to make a suitable margin between the scores of positive and negative triples. However, this loss can not ensure ideal low scores for the positive triplets and high scores for the negative triplets, which is not beneficial for knowledge completion tasks. In this paper, we present a double limit scoring loss to separately set upper bound for correct triplets and lower bound for incorrect triplets, which provides more effective and flexible optimization for knowledge graph embedding. Upon the presented loss framework, we present several knowledge graph embedding models including TransE-SS, TransH-SS, TransD-SS, ProjE-SS and ComplEx-SS. The experimental results on link prediction and triplet classification show that our proposed models have the significant improvement compared to state-of-the-art baselines.

Discrete network embedding emerged recently as a new direction of network representation learning. Compared with traditional network embedding models, discrete network embedding aims to compress model size and accelerate model inference by learning a set of short binary codes for network vertices. However, existing discrete network embedding methods usually assume that the network structures (e.g., edge weights) are readily available. In real-world scenarios such as social networks, sometimes it is impossible to collect explicit network structure information and it usually needs to be inferred from implicit data such as information cascades in the networks. To address this issue, we present an end-to-end discrete network embedding model for latent networks DELN that can learn binary representations from underlying information cascades. The essential idea is to infer a latent Weisfeiler-Lehman proximity matrix that captures node dependence based on information cascades and then to factorize the latent Weisfiler-Lehman matrix under the binary node representation constraint. Since the learning problem is a mixed integer optimization problem, an efficient maximal likelihood estimation based cyclic coordinate descent (MLE-CCD) algorithm is used as the solution. Experiments on real-world datasets show that the proposed model outperforms the state-of-the-art network embedding methods.

Network embedding is a promising topic that maps the vertices to the latent space while keeps the structural proximity in the original space. The network embedding task is difficult since the network vertices have no specific time or space orders. Models that used to extract information from images and texts with regular space or time structures can not be directly applied in network heading. The key feature of network embedding methods should be further exploited. Previous network embedding reviews mainly focus on the models and algorithms used in different methods. In this survey, we review the network embedding works in the stochastic perspective either in data side or model side. Roughly, the network embedding methods fall into three main categories: matrix based methods, random walk based methods and aggregated based methods. We focus on the applications of stochastic models in solving the challenges of network embedding in data processing and modeling following the line of the three categories.

Deep neural networks (DNNs) has drawn considerable attention in recent years as a result of their remarkable performace in many visual and speech recognition assignments. As the scale of tasks that need to solve is increasingly big, the networks used also become wider and deeper, resulting in millions or even billions of parameters needed. Deep and wide networks with large number of parameters bring many problems, including memory requirement, computation cost and overfitting, which severely hinder the application of DNNs in practice. Therefore, a natural thought is to train sparse networks with less parameters and float operators while maintaining comparable performance. During past few years, a mass of research has been proposed in this area. In this paper, we survey sparsity-promoting techniques in DNNs proposed in recent years. These approaches are roughly divided into three categories, including pruning, randomly reducing the complexity and optimizing with sparse regularizer. Pruning techniques will be introduced first and others will be described in the following section. For each kind of methods, we present approaches in this category, strengths and drawbacks. In the final, we will discuss the relationship of these three categories of methods.

How to solve the classification problem with only label proportions has recently drawn increasing attention in the machine learning field. In this paper, we propose an ensemble learning strategy to deal with the learning problem with label proportions (LLP). In detail, we first give a loss function based on different weights for LLP, and then construct the corresponding weak classifier, at the same time, estimate its conditional probabilities by a standard logistic function. At last, by introducing the maximum likelihood estimation, we propose a new anyboost learning system for LLP (called Adaboost-LLP). Unlike traditional methods, our method does not make any restrictive assumptions on training set; at the same time, compared with alter- SVM, Adaboost-LLP exploits more extra weight information and uses multiple weak classifiers that can be solved efficiently to combine a strong classifier. All experiments show that our method outperforms the existing methods in both accuracy and training time.

Feature selection aims to select a subset of features from high-dimensional data according to a predefined selecting criterion. Sparse learning has been proven to be a powerful technique in feature selection. Sparse regularizer, as a key component of sparse learning, has been studied for several years. Although convex regularizers have been used in many works, there are some cases where nonconvex regularizers outperform convex regularizers. To make the process of selecting relevant features more effective, we propose a novel nonconvex sparse metric on matrices as the sparsity regularization in this paper. The new nonconvex regularizer could be written as the difference of the $\ell _{2,1}$ norm and the Frobenius ( $\ell _{2,2}$ ) norm, which is named the $\ell _{2,1-2}$ . To find the solution of the resulting nonconvex formula, we design an iterative algorithm in the framework of ConCave-Convex Procedure (CCCP) and prove its strong global convergence. An adopted alternating direction method of multipliers is embedded to solve the sequence of convex subproblems in CCCP efficiently. Using the scaled cluster indictors of data points as pseudolabels, we also apply $\ell _{2,1-2}$ to the unsupervised case. To the best of our knowledge, it is the first work considering nonconvex regularization for matrices in the unsupervised learning scenario. Numerical experiments are performed on real-world data sets to demonstrate the effectiveness of the proposed method.

Ordinal regression has received increasing interest in the past years. It aims to classify patterns by an ordinal scale. With the the explosive growth of data, the method of SVM with ordinal partitioning called SVMOP highlights its advantages due to its convenience of dealing with large scale data. However, the method of SVMOP for ordinal regression has not been exploited much. As we know, the costs should be different when dealing with mislabeled samples and how to use them plays a dominant role in model building. However, L2-loss which could enlarge the cost sensitivity has not been applied into SVM ordinal partition yet. In this paper, we propose the method of SVMOP with L2-loss for ordinal regression. Numerical results show that our approach outperforms the method of SVMOP with L1-loss and other ordianl regression models.

Clustering has been widely used in data analysis. A majority of existing clustering approaches assume that the number of clusters is given in advance. Recently, a novel clustering framework is proposed which can automatically learn the number of clusters from training data. Based on these works, we propose a nonsmooth penalized clustering model via ℓp(0p-norm-based regularization to control the tradeoff between the model fit and the number of clusters. We theoretically prove that the new model can guarantee the sparseness of cluster centers. To increase its practicality for practical use, we adhere to an easy-to-compute criterion and follow a strategy to narrow down the search interval of cross validation. To address the nonsmoothness and nonconvexness of the cost function, we propose a simple smoothing trust region algorithm and present its convergent and computational complexity analysis. Numerical studies on both simulated and practical data sets provide support to our theoretical results and demonstrate the advantages of our new method.

Recently, learning from label proportions (LLPs), which seeks generalized instance-level predictors merely based on bag-level label proportions, has attracted widespread interest. However, due to its weak label scenario, LLP usually falls into a transductive learning framework accounting for an intractable combinatorial optimization issue. In this paper, we propose a brand new algorithm, called LLPs via nonparallel support vector machine (LLP-NPSVM), to facilitate this dilemma. To harness satisfactory data adaption, instead of transductive learning fashion, our scheme determined instance labels according to two nonparallel hyper-planes under the supervision of label proportion information. In a geometrical view, our approach can be interpreted as an alternative competitive method benefiting from large margin clustering. In practice, LLP-NPSVM can be efficiently addressed by applying two fast sequential minimal optimization paths iteratively. To rationally support the effectiveness of our method, finite termination and monotonic decrease of the proposed LLP-NPSVM procedure were essentially analyzed. Various experiments demonstrated our algorithm enjoys rapid convergence and robust numerical stability, along with best accuracies among several recently developed methods in most cases.

Ordinal regression is a supervised learning problem where training samples are labeled by an ordinal scale. The ordering relation and nonmetric property of the label set distinguish it from the multiclass classification and metric regression. To better exploit the inherent structure in the label and benefit from the hidden information in data distribution, we propose a novel ordinal regression model, which is named as nonparallel support vector ordinal regression (NPSVOR) to emphasis the utilization of nonparallel proximal hyperplanes. The new model constructs a hyperplane for each rank such that the patterns of this rank lie in the close proximity while maintaining clear separation with the other ranks. Since the learning of hyperplanes can be carried out independently, NPSVOR can be trained in parallel. Furthermore, we design an efficient solver at the same time for training the hyperplanes in NPSVOR based on the alternating direction method of multipliers. Extensive experimentation demonstrates that NPSVOR yields a large and statistically significant improvement in terms of generalization performance and training speed against nine baselines.

Handling data classification and regression problems through linear hyperplane is a naive and simple idea. In this paper, inspired by the idea of multiple criteria linear programs (MCLP) and multiple criteria quadratic programs (MCQP), we proposed a novel method for binary classification and regres sion problem. There are two main advantages for the proposed approach. One is that both of these two models guarantee the existence of feasible solutions when the model parameters were chosen properly. The other is that nonlinear patterns could be handled and captured by introducing kernel function into MCLP framework with a more natural way than previous work. Various classical approaches and datasets were evaluated in our experiments, and the result on both toy and real world data demonstrate the correctness and effectiveness of our proposed methods.