As one of the most common ways for user authentication, Personal Identification Number (PIN), due to its simplicity and convenience, has suffered from plenty of side-channel attacks, which pose a severe threat to people’s privacy and property. The success of existing attacks is usually built upon the premise of no occlusion between the attacker and the victim’s hand gesture, but it increases the difficulty of launching the attack and the possibility of exposure. To overcome such limitation, we propose ArmSpy++, an improved video-assisted PIN inference attack built upon our previous research, ArmSpy. Specifically, ArmSpy++ employs new modules to leverage more features like the keystroke-induced elbow bending, wrist speed variation, and the spatial relationship between different arm joints, to correctly detect Keystrokes. ArmSpy++ delves into the perspective relationship and natural typing habits to ensure a high success rate of PIN inference. We also re-designed the inferred PIN pattern coordination mechanism to accurately deduce the PINs. By using a pre-trained HigherHRNet model for posture estimation ArmSpy++ eliminates the necessity of additional training. The extensive experiments demonstrate that ArmSpy++ can achieve over 83.1% average accuracy with 3 attempts and even 92.5% for some victims, indicating the severity of the threat posed by ArmSpy++.

In this paper, we introduce a fresh perspective on the challenges of credit assignment and policy evaluation. First, we delve into the nuances of eligibility traces and explore instances where their updates may result in unexpected credit assignment to preceding states. From this investigation emerges the concept of a novel value function, which we refer to as the ????????????? ????? ????????. Unlike traditional state value functions, bidirectional value functions account for both future expected returns (rewards anticipated from the current state onward) and past expected returns (cumulative rewards from the episode's start to the present). We derive principled update equations to learn this value function and, through experimentation, demonstrate its efficacy in enhancing the process of policy evaluation. In particular, our results indicate that the proposed learning approach can, in certain challenging contexts, perform policy evaluation more rapidly than TD(λ)–a method that learns forward value functions, v^π, ????????. Overall, our findings present a new perspective on eligibility traces and potential advantages associated with the novel value function it inspires, especially for policy evaluation.

The two-factor authentication (2FA) has become pervasive as the mobile devices become prevalent. Existing 2FA solutions usually require some form of user involvement, which could severely affect user experience and bring extra burdens to users. In this work, we propose a secure 2FA that utilizes the individual acoustic fingerprint of the speaker/microphone on enrolled device as the second proof. The main idea behind our system is to use both magnitude and phase fingerprints derived from the frequency response of the enrolled device by emitting acoustic beep signals alternately from both enrolled and login devices and receiving their direct arrivals for 2FA. Given the input microphone samplings, our system designs an arrival time detection scheme to accurately identify the beginning point of the beep signal from the received signal. To achieve a robust authentication, we develop a new distance mitigation scheme to eliminate the impact of transmission distances from the sound propagation model for extracting stable fingerprint in both magnitude and phase domain. Our device authentication component then calculates a weighted correlation value between the device profile and fingerprints extracted from run-time measurements to conduct the device authentication for 2FA. Moreover, to thwart the possible co-located attacks, our proximity detection component further makes the enrolled phone to generate an active random vibration signal by its built-in motor, and then matches the signal received by the microphone of login device with the signal received by the accelerometer of enrolled phone to verify the proximity of two devices. Our experimental results show that our proposed system is accurate and robust to various attacks across different scenarios and device models.

Despite recent advances in modern machine learning algorithms, the opaqueness of their underlying mechanisms continues to be an obstacle in adoption. To instill confidence and trust in artificial intelligence (AI) systems, explainable AI (XAI) has emerged as a response to improve modern machine learning algorithms' explainability. Inductive logic programming (ILP), a subfield of symbolic AI, plays a promising role in generating interpretable explanations because of its intuitive logic-driven framework. ILP effectively leverages abductive reasoning to generate explainable first-order clausal theories from examples and background knowledge. However, several challenges in developing methods inspired by ILP need to be addressed for their successful application in practice. For example, the existing ILP systems often have a vast solution space, and the induced solutions are very sensitive to noises and disturbances. This survey paper summarizes the recent advances in ILP and a discussion of statistical relational learning (SRL) and neural-symbolic algorithms, which offer synergistic views to ILP. Following a critical review of the recent advances, we delineate observed challenges and highlight potential avenues of further ILP-motivated research toward developing self-explanatory AI systems.

Ensuring reliability in large-scale cloud systems like Microsoft 365 is crucial. Cloud failures, such as disk and node failure, threaten service reliability, causing service interruptions and financial loss. Existing works focus on failure prediction and proactively taking action before failures happen. However, they suffer from poor data quality, like data missing in model training and prediction, which limits performance. In this paper, we focus on enhancing data quality through data imputation by the proposed Diffusion+, a sample-efficient diffusion model, to impute the missing data efficiently conditioned on the observed data. Experiments with industrial datasets and application practice show that our model contributes to improving the performance of downstream failure prediction.

We present a mean-variance policy iteration (MVPI) framework for risk-averse control in a discounted infinite horizon MDP optimizing the variance of a per-step reward random variable. MVPI enjoys great flexibility in that any policy evaluation method and risk-neutral control method can be dropped in for risk-averse control off the shelf, in both on- and off-policy settings. This flexibility reduces the gap between risk-neutral control and risk-averse control and is achieved by working on a novel augmented MDP directly. We propose risk-averse TD3 as an example instantiating MVPI, which outperforms vanilla TD3 and many previous risk-averse control methods in challenging Mujoco robot simulation tasks under a risk-aware performance metric. This risk-averse TD3 is the first to introduce deterministic policies and off-policy learning into risk-averse reinforcement learning, both of which are key to the performance boost we show in Mujoco domains.

With the development of modern life, all kinds of social activities have become more frequent. These social activities are often attended by a wide range of people, which puts forward high requirements for effective management and ensures the safety of the people involved in the activities. As an effective auxiliary measure, crowd understanding and analysis has been concerned by more and more researchers. The basic idea is to extract the key information from video sequences/images, and use digital image processing technology to study and analyse the behaviour characteristics and patterns of people in the region of interest. Currently, it has a wide range of applications in the fields of economy, public security, and so forth. This Special Issue aims to introduce the latest works in crowd understanding and analysis, and proposes new theory and approaches to solve the existing problems. The Special Issue received a number of submissions from researchers in the field. Finally, 27 papers were accepted after careful peer reviews and revisions. The accepted papers can be broadly divided into six sections according to the types of tasks, including Section 1: Crowd behaviour detection and recognition, Section 2: Crowd counting, Section 3: Object detection and recognition, and Section 4: Object tracking, and Section 5: Other tasks. “Human behaviour recognition with mid-level representations for crowd understanding and analysis” of Sun et al. uses mid-level semantic concepts to represent human actions from videos and argues that these semantic attributes enable the construction of more descriptive methods for human action recognition. The idea is verified on three challenging datasets, and the experimental results demonstrate that their method achieves better results than the baseline methods on human action recognition. “Class structure-aware adversarial loss for cross-domain human action recognition” of Chen et al. proposes a class structure-aware adversarial loss, which aims to address the issue that the existing adversarial-based approaches ignore the underlying coherence of class structure across domains. This paper incorporates category information into the adversarial learning branch to capture the fine-grained alignment of each class, effectively avoiding the false mix-up of samples from different categories in the embedding space. Experiments show significant improvement compared to the baseline. “Dual-view 3D human pose estimation without camera parameters for action recognition” of Liu et al. proposes a dual-view single-person 3D pose estimation method without camera parameters. This method first uses the 2D pose estimation network to estimate the 2D joint point coordinates from two images with different views, and then inputs them into the 3D regression network to generate the final 3D joint point coordinates. Experiments show that this method is effective. “Deep social force network for anomaly event detection” of Yang et al. develops a deep social force network by exploiting both social force extraction and deep motion coding. This network can discover the interaction force of particles to learn the deep social force features. The experiments on UCF-Crime and ShanghaiTech datasets demonstrate that this method can predict the temporal localization of anomaly events and outperform the state-of-the-art methods. “Anomaly detection in video sequences: a benchmark and computational model” of Wan et al. contributes a new Large-scale Anomaly Detection (LAD) dataset as the benchmark for anomaly detection in video sequences. This paper formulates anomaly detection as a fully supervised learning problem and proposes a multi-task deep neural network to solve it. Experimental results show that the proposed method outperforms state-of-the-art anomaly detection competitors on the proposed dataset and other public datasets. “Behaviour detection in crowded classroom scenes via enhancing features robust to scale and perspective variations” of Liu et al. proposes two modules to tackle the large variations of humans in scale and pose perspective, namely Scale Attention Aggregation module and RoI Spatial Transformation module. Besides, this paper constructs a new classroom human behaviour detection dataset with 1500 images based on a third-person perspective. The proposed method is verified on the proposed dataset. Experimental results demonstrate the effectiveness of the proposed method with better mAP values. “Crowd activity recognition in live video streaming via 3D-ResNet and region graph convolution network” of Kang et al. presents a crowd activity recognition method to identify and supervise the crowd activity from live videos. The method utilizes 3D-ResNet and ReGCN to extract deep spatiotemporal features and the correlation or external knowledge between crowd content, respectively. In addition to the above work, the authors construct a real-world video dataset called BJUT-CAD that includes eight kinds of crowd activity videos collected from live video websites. Experiments on BJUT-CAD and CAE datasets verify the effectiveness of this work. “Latent label mining for group activity recognition in basketball videos” of Wu et al. proposes a latent label mining strategy for group activity recognition in basketball videos. This paper aims at mining the latent labels of motion patterns from the frames and further combining two levels of supervision signal to obtain effective spatio-temporal representation. Experimental results demonstrate that the proposed algorithm achieves state-of-the-art performance. “A deep learning method for video-based action recognition” of Zhang et al. aims to address a key issue, that is, how to convert spatial and temporal information into an effective representation to infer actions. This paper employs boundary compensation on the basis of a deep neural network to achieve action proposal. Based on the resultant action proposals, a two-stream network with a spatio-temporal structure is adopted for the action recognition task. The experimental results show the competitive performance of the proposed method over the state-of-the-art methods. “MSR-FAN: Multi-scale residual feature-aware network for crowd counting” of Zhao et al. proposes a framework that combines the multi-scale features using multiple receptive field sizes and learns the feature-aware information on each image. This method effectively alleviates perspective distortion and the varying scales in congested scene images, which helps the algorithm crowd counting correctly. Experiment results on benchmark datasets indicate that the proposed approach outperforms the existing competitors. “MFP-Net: Multi-scale feature pyramid network for crowd counting” of Lei et al. introduces a feature pyramid fusion module and a feature attention-aware module. Two modules can extract different levels of fine-grained information; local and global context information, respectively. It enhances the correlation of different features and improves robustness to background noise effectively. Experiments show that the proposed method not only provides better crowd counting results than comparative models, but also requires fewer parameters. “Multi-level features extraction network with gating mechanism for crowd counting” of Zeng et al. designs a novel crowd counting model, which integrates multi-level information from multiple levels, such as appearance, scale, and context. To avoid interference from confusing information, a simple and effective multi-channel gated unit is proposed to adaptively select features at different levels of the network. Extensive experiments and evaluations clearly illustrate that the approach is superior. “Learn from object counting: crowd counting with meta-learning” of Zan et al. develops an efficient algorithm to extract the meta-information via utilizing object counting data in few-shot scenes. This method successfully explores the shared information between the crowd counting task and the object counting task, thus improving the performance and convergence rate of the crowd counting task. Comprehensive experiments on two datasets verify the effectiveness of the proposed approach. “Crowd estimation using key-point matching with support vector regression” of Ekanayake et al. proposes a novel key-point-based moving object detection in noisy backgrounds. This work mainly uses the key-point matching of continuous frames and optical flow density to identify moving objects in video sequences. The moving blobs are then generated via morphological operations. The experiments are compared with recent regression-based and CNN methods, and it is verified that the proposed work is superior. “A novel face recognition method based on fusion of LBP and HOG” of Chen et al. proposes an improved fusion local feature extraction algorithm called CS-NWALBP+HOG. This study not only smoothens noise sensitivity of the LBP operator, but also reduces the original computational complexity, as well as strengthens the description ability for image gradient direction information. Several experiments eventually demonstrate that the designed algorithm shows more robust performance under complex illumination conditions. “Multi-view intrinsic low-rank representation for robust face recognition and clustering” of Shen et al. considers the problem that the most existing methods ignore the specific local structure of different views. To address this issue, the paper proposes a multi-view low-rank representation method which exploits both intrinsic relationships and specific local structures of different views simultaneously. Experiments on several datasets demonstrate the effectiveness of this method in classification and clustering. “Multi-dimensional weighted cross-attention network in crowded scenes” of Xie et al. proposes an end-to-end anchor-free network, namely Multidimensional Weighted Cross-Attention Network, which can perform real-time human detection in crowded scenes. The designed model does not require manual intervention, and reduces the sizeable computational resource cost due to the anchor boxes mapping during the training process. Experiments reveal that the improved strategy achieves state-of-the-art results in the anchor-free methods. “Part-level attention networks for cross-domain person re-identification” of Zhao et al. uses the diversified spatial semantic feature in pixel-level learning in the target domain to improve the generality and adaptability of the model. Combined with partial branches, this method proves that it is effective to add an attention cascade module to the backbone network. Experiments indicate that the proposed approach has better recognition ability and robustness in cross-domain aspect. “MFNet-LE: Multilevel fusion network with Laplacian embedding for face presentation attacks detection” of Niu et al. proposes a face presentation attack detection method by incorporating a multilevel fusion structure and Laplacian loss into shallow CNNs. This allows the proposed model to learn more discriminative features under the joint supervision of softmax and Laplacian loss, which improves the detection ability. Experiments demonstrate the effectiveness of the proposed method. “Real-time automatic helmet detection of motorcyclists in urban traffic using improved YOLOv5 detector” of Jia et al. presents an automatic helmet detection that contains two steps. This work first utilizes improved YOLOv5 detector to detect motorcycles. Then, the motorcycle detected in the above step is input, and the improved YOLOv5 detector is used again to detect whether the rider is wearing a helmet. The proposed method is evaluated on constructed dataset, and the results show that it is superior to other detection methods. “Multi-label learning based target detecting from multi-frame data” of Mei et al. regards target detection from time series data as a multi-label problem to design the model. In this method, a background subtraction tracker is presented to track the slightly moving object in videos, which are based on Gaussian mixture model background subtraction and integral image. Experimental results show that the proposed method attains better performance. “Contrastive learning of graph encoder for accelerating pedestrian trajectory prediction training” of Yao et al. proposes a graph contrastive accelerating encoder. It accelerates the pedestrian trajectory prediction training process of spatiotemporal graph transformer networks. This method makes the pedestrian trajectory prediction error the lowest in the obviously early training steps, and makes the final performance reach the state-of-the-art level. “Multiple object tracking based on multi-task learning with strip attention” of Song et al. believes that it is difficult to strike a balance between accuracy and efficiency by embedding the re-identification (re-ID) model into the target tracking task. To enhance the overall tracking performance, a one-shot multiple object tracking is proposed based on multi-task learning, which contains two homogeneous branches of object detection and re-ID. By the fine-grained features extraction in pedestrian recognition, it benefits overall tracking framework improvement in both speed and robustness. The experiments show that the proposed method attains superior performance in more evaluation metrics. “Cross-modal semantic correlation learning by Bi-CNN network” of Wang et al. presents a novel cross modal retrieval framework, which integrates feature learning and latent space embedding. It aims to generate specific representations consistent with cross-modal tasks. It helps to reduce the differences in the distribution of categories in different modalities. Experiments on three real-word datasets show that the proposed work is superior to the popular methods. “Adaptive colour restoration and detail retention for image enhancement” of He et al. aims to overcome the problem of colour distortion caused by low illumination and fog. Considering the issue, this paper develops a multi-channel fusion-based adaptive image colour restoration method. To generate human-consistent observations, the detailed retention-based method is applied to enhance the details. Experiments demonstrate that the results are effective and outperform the compared methods both in visual and objective evaluations. “Image encryption algorithm for crowd data based on a new hyperchaotic system and Bernstein polynomial” of Jiang et al. designs a new two-dimensional chaotic system with hyperchaotic behaviour based on the Chebyshev system and the infinite collapse system. To protect the crowd image data, an image cryptosystem combined with the SVD and Bernstein polynomial is proposed. Security analyses indicate that this method has higher encryption efficiency and the visual quality of steganography image can reach 39 dB. “CA-PMG: Channel attention and progressive multi-granularity training network for fine-grained visual classification” of Zhao et al. designs a framework for visual classification for the subtle intra-class object variations. This model can be trained efficiently in an end-to-end manner without bounding box or part annotations. Extensive experiments on three challenging fine-grained datasets demonstrate that the approach obtains state-of-the-art performance. The papers selected in this Special Issue highlight the extensive study of crowd understanding and analysis. It is hoped that these papers can play a role in promoting theoretical research. Meanwhile, there are many challenges in this field that need to be further studied, such as the robustness of algorithms in cross-scenarios, the interaction between groups and individuals in different scenarios, and so forth. In addition, key problems such as the running time of the algorithm also need to be considered. These works will be helpful when applied to research in the real world. Lead Guest Editor Professor Qi Wang, Northwestern Polytechnical University, Xi'an, China Guest Editors Assistant Professor Bo Liu, Auburn University, USA Dr. Jianzhe Lin, University of British Columbia, Canada

We present GradientDICE for estimating the density ratio between the state distribution of the target policy and the sampling distribution in off-policy reinforcement learning. GradientDICE fixes several problems of GenDICE (Zhang et al., 2020), the current state-of-the-art for estimating such density ratios. Namely, the optimization problem in GenDICE is not a convex-concave saddle-point problem once nonlinearity in optimization variable parameterization is introduced to ensure positivity, so primal-dual algorithms are not guaranteed to find the desired solution. However, such nonlinearity is essential to ensure the consistency of GenDICE even with a tabular representation. This is a fundamental contradiction, resulting from GenDICE’s original formulation of the optimization problem. In GradientDICE, we optimize a different objective from GenDICE by using the Perron-Frobenius theorem and eliminating GenDICE’s use of divergence, such that nonlinearity in parameterization is not necessary for GradientDICE, which is provably convergent under linear function approximation.

We present the first provably convergent two-timescale off-policy actor-critic algorithm (COF-PAC) with function approximation. Key to COF-PAC is the introduction of a new critic, the emphasis critic, which is trained via Gradient Emphasis Learning (GEM), a novel combination of the key ideas of Gradient Temporal Difference Learning and Emphatic Temporal Difference Learning. With the help of the emphasis critic and the canonical value function critic, we show convergence for COF-PAC, where the critics are linear and the actor can be nonlinear.

Deep reinforcement learning (DRL) has gained great success by learning directly from high-dimensional sensory inputs, yet is notorious for the lack of interpretability. Interpretability of the subtasks is critical in hierarchical decision-making as it increases the transparency of black-box-style DRL approach and helps the RL practitioners to understand the high-level behavior of the system better. In this paper, we introduce symbolic planning into DRL and propose a framework of Symbolic Deep Reinforcement Learning (SDRL) that can handle both high-dimensional sensory inputs and symbolic planning. The task-level interpretability is enabled by relating symbolic actions to options.This framework features a planner – controller – meta-controller architecture, which takes charge of subtask scheduling, data-driven subtask learning, and subtask evaluation, respectively. The three components cross-fertilize each other and eventually converge to an optimal symbolic plan along with the learned subtasks, bringing together the advantages of long-term planning capability with symbolic knowledge and end-to-end reinforcement learning directly from a high-dimensional sensory input. Experimental results validate the interpretability of subtasks, along with improved data efficiency compared with state-of-the-art approaches.

Deep reinforcement learning (DRL) has gained great success by learning directly from high-dimensional sensory inputs, yet is notorious for the lack of interpretability. Interpretability of the subtasks is critical in hierarchical decision-making as it increases the transparency of black-box-style DRL approach and helps the RL practitioners to understand the high-level behavior of the system better. In this paper, we introduce symbolic planning into DRL and propose a framework of Symbolic Deep Reinforcement Learning (SDRL) that can handle both high-dimensional sensory inputs and symbolic planning. The task-level interpretability is enabled by relating symbolic actions to options. This framework features a planner – controller – meta-controller architecture, which takes charge of subtask scheduling, data-driven subtask learning, and subtask evaluation, respectively. The three components cross-fertilize each other and eventually converge to an optimal symbolic plan along with the learned subtasks, bringing together the advantages of long-term planning capability with symbolic knowledge and end-to-end reinforcement learning directly from a high-dimensional sensory input. Experimental results validate the interpretability of subtasks, along with improved data efficiency compared with state-of-the-art approaches.

Although promising results have been achieved in the restoration of complex illumination images with the Retinex algorithm, there are still some drawbacks in the processing of Retinex. Considering the noise characteristics of complex illumination images, in this study, we propose a novel restoration algorithm for noisy complex illumination, which combines guided adaptive multi‐scale Retinex (GAMSR) and improvement BayesShrink threshold filtering (IBTF) based on double‐density dual‐tree complex wavelet transform (DDDTCWT) domain. Extensive restoration experiments are conducted on three typical types images and the same image with different noises. On the basis of a series of evaluation indexes, we compare our method to those of state‐of‐the‐art algorithms. The results show that (i) SSIM of the proposed IBTF is superior to traditional Bayes threshold method by 15% as the standard variance is 100. (ii) PSNR of the proposed GAMSR enhances 15% to traditional MSR. (iii) The clarity of final results for restoration speeds up three times than that of original images, and the information entropy is improved slightly too. Therefore, the proposed method can effectively enhance the details, edges and textures of the image under complex illumination and noises.

Single image super‐resolution (SISR) has gained great attraction and progress in recent years. Since the SISR is an ill‐posed inverse problem, most researchers are concentrated on making efforts to learn effective and reasonable mapping functions from low‐resolution observation to its potential high‐resolution (HR) counterpart. In this study, the authors have proposed a deep residual refining based pseudo‐multi‐frame network for efficient SISR. A channel‐wise attention mechanism is employed for residual refinement. It can ease residual learning process through explicitly modelling non‐linear dependencies between channels by using global information embedding. Multiple potential HRs from different deconvolutional layers are further artificially learned, and then adaptively fused into final desired HR image. The authors call this strategy as pseudo‐multi‐frame SR. It could make full use of available redundant information possessed in hierarchical layers. They have evaluated the proposed network on several popular benchmark datasets. The experimental results have shown that the two highlights proposed can consistently boost final performance. The proposed network can outperform most of the state‐of‐the‐art methods with acceptable less parameters.

Policy evaluation algorithms are essential to reinforcement learning due to their ability to predict the performance of a policy. However, there are two long-standing issues lying in this prediction problem that need to be tackled: off-policy stability and on-policy efficiency. The conventional temporal difference (TD) algorithm is known to perform very well in the on-policy setting, yet is not off-policy stable. On the other hand, the gradient TD and emphatic TD algorithms are off-policy stable, but are not on-policy efficient. This paper introduces novel algorithms that are both off-policy stable and on-policy efficient by using the oblique projection method. The empirical experimental results on various domains validate the effectiveness of the proposed approach.

Random projection is a popular machine learning algorithm, which can be implemented by neural networks and trained in a very efficient manner. However, the number of features should be large enough when applied to a rather large-scale data set, which results in slow speed in testing procedure and more storage space under some circumstances. Furthermore, some of the features are redundant and even noisy since they are randomly generated, so the performance may be affected by these features. To remedy these problems, an effective feature selection method is introduced to select useful features hierarchically. Specifically, a novel criterion is proposed to select useful neurons for neural networks, which establishes a new way for network architecture design. The testing time and accuracy of the proposed method are improved compared with traditional methods and some variations on both classification and regression tasks. Extensive experiments confirm the effectiveness of the proposed method.

Reinforcement learning and symbolic planning have both been used to build intelligent autonomous agents. Reinforcement learning relies on learning from interactions with real world, which often requires an unfeasibly large amount of experience. Symbolic planning relies on manually crafted symbolic knowledge, which may not be robust to domain uncertainties and changes. In this paper we present a unified framework PEORL that integrates symbolic planning with hierarchical reinforcement learning (HRL) to cope with decision-making in dynamic environment with uncertainties. Symbolic plans are used to guide the agent's task execution and learning, and the learned experience is fed back to symbolic knowledge to improve planning. This method leads to rapid policy search and robust symbolic plans in complex domains. The framework is tested on benchmark domains of HRL.

Risk management in dynamic decision problems is a primary concern in many fields, including financial investment, autonomous driving, and healthcare. The mean-variance function is one of the most widely used objective functions in risk management due to its simplicity and interpretability. Existing algorithms for mean-variance optimization are based on multi-time-scale stochastic approximation, whose learning rate schedules are often hard to tune, and have only asymptotic convergence proof. In this paper, we develop a model-free policy search framework for mean-variance optimization with finite-sample error bound analysis (to local optima). Our starting point is a reformulation of the original mean-variance function with its Fenchel dual, from which we propose a stochastic block coordinate ascent policy search algorithm. Both the asymptotic convergence guarantee of the last iteration's solution and the convergence rate of the randomly picked solution are provided, and their applicability is demonstrated on several benchmark domains.

l2,1-norm is an effective regularization to enforce a simple group sparsity for feature learning. To capture some subtle structures among feature groups, we propose a new regularization called exclusive group l2,1-norm. It enforces the sparsity at the intra-group level by using l2,1-norm, while encourages the selected features to distribute in different groups by using l2 norm at the inter-group level. The proposed exclusivegroup l2,1-norm is capable of eliminating the feature correlationsin the context of feature selection, if highly correlated features are collected in the same groups. To solve the generic exclusive group l2,1-norm regularized problems, we propose an efficient iterative re-weighting algorithm and provide a rigorous convergence analysis. Experiment results on real world datasets demonstrate the effectiveness of the proposed new regularization and algorithm.

It is meaningful to design a strategy to roughly localize mobile phones without a GPS by exploiting existing conditions and devices especially in environments without GPS availability (e.g., tunnels, subway stations, etc.). The availability of Bluetooth devices for most phones and the existence of a number of GPS equipped phones in a crowd of phone users enable us to design a Bluetooth aided mobile phone localization strategy. With the position of GPS equipped phones as beacons, and with the Bluetooth connection between neighbor phones as proximity constraints, we formulate the problem into an inequality problem defined on the Bluetooth network. A recurrent neural network is developed to solve the problem distributively in real time. The convergence of the neural network and the solution feasibility to the defined problem are both theoretically proven. The hardware implementation architecture of the proposed neural network is also given in this article. As applications, rough localizations of drivers in a tunnel and localization of customers in a supermarket are explored and simulated. Simulations demonstrate the effectiveness of the proposed method.

The winner-take-all (WTA) competition is widely observed in both inanimate and biological media and society. Many mathematical models are proposed to describe the phenomena discovered in different fields. These models are capable of demonstrating the WTA competition. However, they are often very complicated due to the compromise with experimental realities in the particular fields; it is often difficult to explain the underlying mechanism of such a competition from the perspective of feedback based on those sophisticate models. In this paper, we make steps in that direction and present a simple model, which produces the WTA competition by taking advantage of selective positive-negative feedback through the interaction of neurons via p-norm. Compared to existing models, this model has an explicit explanation of the competition mechanism. The ultimate convergence behavior of this model is proven analytically. The convergence rate is discussed and simulations are conducted in both static and dynamic competition scenarios. Both theoretical and numerical results validate the effectiveness of the dynamic equation in describing the nonlinear phenomena of WTA competition.

We present a novel $l_1$ regularized off-policy convergent TD-learning method (termed RO-TD), which is able to learn sparse representations of value functions with low computational complexity. The algorithmic framework underlying RO-TD integrates two key ideas: off-policy convergent gradient TD methods, such as TDC, and a convex-concave saddle-point formulation of non-smooth convex optimization, which enables first-order solvers and feature selection using online convex regularization. A detailed theoretical and experimental analysis of RO-TD is presented. A variety of experiments are presented to illustrate the off-policy convergence, sparse feature selection capability and low computational cost of the RO-TD algorithm.

This paper explores links between basis construction methods in Markov decision processes and power series expansions of value functions. This perspective provides a useful framework to analyze properties of existing bases, as well as provides insight into constructing more effective bases. Krylov and Bellman error bases are based on the Neumann series expansion. These bases incur very large initial Bellman errors, and can converge rather slowly as the discount factor approaches unity. The Laurent series expansion, which relates discounted and average-reward formulations, provides both an explanation for this slow convergence as well as suggests a way to construct more efficient basis representations. The first two terms in the Laurent series represent the scaled average-reward and the average-adjusted sum of rewards, and subsequent terms expand the discounted value function using powers of a generalized inverse called the Drazin (or group inverse) of a singular matrix derived from the transition matrix. Experiments show that Drazin bases converge considerably more quickly than several other bases, particularly for large values of the discount factor. An incremental variant of Drazin bases called Bellman average-reward bases (BARBs) is described, which provides some of the same benefits at lower computational cost.