Dialogue policy is a core module in pipeline dialogue systems as it drives conversation generation. Personalized dialogue policies aim to equip chatbots with tailored personalities, making them behave like real users, providing more accurate action responses, and improving the anthropomorphic capabilities of personal assistants. Yet existing dialogue policy approaches often overlook individual personalities because obtaining explicit user profiles is costly and time-consuming. In this article, we propose a personalized dialogue policy learning framework, named PDL. It dynamically learns implicit user profiles from successful dialogue trajectories. Specifically, we collect a lot of success histories from human–computer interactions to extract sequences of user belief states and agent actions. The extracted sequences are processed via a loop clipping operation and modeled with an autoregressive transformer to mimic human analytical behavior. After that, a new user’s latent personalized preferences are predicted based on the autoregressive transformer model. The personalized preferences are employed to implement dialogue policies via three categories of reinforcement learning algorithms, including value-based approaches, policy-based approaches, and model-based approaches. The experiments are conducted on three different task-oriented dialogue datasets, and the results show that the proposed PDL framework achieves state-of-the-art results compared to other comparative approaches.

Abstract Training a task-oriented dialogue policy using deep reinforcement learning is promising but requires extensive environment exploration. The amount of wasted invalid exploration makes policy learning inefficient. In this paper, we define and argue that dead-end states are important reasons for invalid exploration. When a conversation enters a dead-end state, regardless of the actions taken afterward, it will continue in a dead-end trajectory until the agent reaches a termination state or maximum turn. We propose a Dead-end Detection and Resurrection (DDR) method that detects dead-end states in an efficient manner and provides a rescue action to guide and correct the exploration direction. To prevent dialogue policies from repeating errors, DDR also performs dialogue data augmentation by adding relevant experiences that include dead-end states and penalties into the experience pool. We first validate the dead-end detection reliability and then demonstrate the effectiveness and generality of the method across various domains through experiments on four public dialogue datasets.

Reinforcement learning (RL) has emerged as a key technique for designing dialogue policies. However, action space inflation in dialogue tasks has led to a heavy decision burden and incoherence problems for dialogue policies. In this paper, we propose a novel decomposed deep Q-network (D2Q) that exploits the natural structure of dialogue actions to perform decomposition on Q-function, realizing efficient and coherent dialogue policy learning. Instead of directly evaluating the Q-function, it consists of two separate estimators, one for the abstract action-value functions and the other for the specific action-value functions, both sharing a common feature layer. The abstract action-value function determines the speech act of the system action, while the specific action-value function focuses on the concrete action. This structure establishes a logical relationship between the user and the system on speech actions, avoiding the problem of incoherence. Moreover, the abstract action-value function shields unreasonable specific actions in the inflated action space, reducing the decision complexity. Our results show that the problem of incoherence is prevalent in existing approaches, which significantly impacts the efficiency and quality of dialogue policy learning. Our D2Q architecture alleviates this problem and performs significantly better than competitive baselines in both evaluated and human experiments. Further experiments validate the generality of our method. It can be easily extended to other RL-based dialogue policy approaches.

Federated Learning (FL) has been widely used for distributed machine learning in edge computing. In FL, the model parameters are iteratively aggregated from the clients to a central server, which is inclined to be the communication bottleneck and single point of failure. To solve these drawbacks, hierarchical model training frameworks like Hierarchical Federated Learning (HFL) and E-Tree learning have been proposed. One of the most challenging problems in the hierarchical model training framework is optimizing the aggregation frequencies of the edge devices at various levels. Because, in an edge computing environment, heterogeneity in the resource can introduce synchronization delays caused by waiting for slow workers and significantly impact the training performance. This paper tackles the problem with weak synchronization where edge devices on the same level have different frequencies on local updates and/or model aggregations. Existing works based on weak synchronization lack solutions to quantitatively determine the aggregation frequencies of each edge device. Thus, we propose a resource-based aggregation frequency controlling method, termed RAF, which determines the optimal aggregation frequencies of edge devices to minimize the loss function according to heterogeneous resources. Our proposed method can alleviate the waiting time and fully utilize the resources of the edge devices. Besides, RAF dynamically adjusts the aggregation frequencies at different phases during the model training to achieve fast convergence speed and high accuracy. We evaluated the performance of RAF via extensive experiments with real datasets on our self-developed edge computing testbed. Evaluation results demonstrate that RAF outperforms the benchmark approaches in terms of learning accuracy and convergence speed.

Spatial crowdsourcing as a promising computing paradigm has received significant attention recently. A fundamental issue of spatial crowdsourcing is online task assignment, i.e., the platform must make decisions immediately (assign or reject) for newly arriving objects (tasks or workers). Previous studies mostly focus on the rejection-aware assignment, which rarely considers non-rejection assignment for new arrival objects. To solve this new allocation model, in this paper, we first formulate a novel problem, namely Online Non-rejection aware Task Assignment (ONRTA) in spatial crowdsourcing, where an object cannot be rejected by the platform as long as there is a neighbor that satisfies the matching constraint with it. Then, we develop a non-rejection threshold-based random algorithm ONRTA-RT under the adversarial order model while obtaining a theoretical bound on the competitive ratio. More importantly, we consider a more natural random order model and propose a two-stage-based non-rejection aware task assignment approach, ONRTA-Base, which achieves a competitive ratio of$\frac{1}{4}$. Based on this framework, we further devise two non-rejection assignment approaches, ONRTA-OP and ONRTA-Greedy, which are more effective and run faster with a competitive ratio of$\frac{1}{4}$and$\frac{1}{8}$, respectively. Finally, experiments on synthetic and real datasets demonstrate that our proposed methods outperform the representative methods.

Content-Defined Chunking(CDC) is the key stage of data deduplication since it has a significant impact on deduplication system’s throughput and deduplication efficiency. However, existing CDC algorithms suffer from high computation overhead, weak stability, and poor ability to handle low-entropy strings. In this paper, we propose UltraCDC, a fast and stable, high-efficiency deal with low-entropy strings, CDC algorithm for deduplication-based storage systems. There are four key techniques behind UltraCDC, namely, rolling compute boundary conditions, skipping sub-minimum chunk size, normalized chunking, and jumping to detect low-entropy strings. Using a sliding window to rolling compute boundary conditions not only accelerates the chunking stage but also makes it more resistant to boundary shift, the two techniques of skipping sub-minimum chunk size and normalized chunking can complement each other to speed up chunking without sacrificing deduplication ratio too much, and the jumping detection can detect more low-entropy strings than AE-opt2 without affecting chunking speed. We implemented UltraCDC in Destor, and the experimental results show that using the above four techniques, chunking speed is 1.5–10× faster than the state-of-the-art CDC approaches, while deduplication ratio is comparable or even higher than the classic Rabin-base CDC. In terms of the capability to detect low-entropy strings, UltraCDC is a CDC approach with the highest ability to detect low-entropy strings, 102× and 2× higher than Rabin-based CDC and AE-opt2, respectively.

It has been proved that three-dimensional (3D) convolutional kernel can effectively capture local features in the spatiotemporal range of videos, leading to impressive results of various models in video-related tasks. With the introduction of Transformer and the rise of self-attention mechanism, more self-attention models have been used on video representation learning recently. However, there exist limitations of local perception and self-attention operations respectively in both two types of models. Inspired by the global context network (GCNet), we take advantages of both 3D convolution and self-attention mechanism to design a novel operator called the GC-Conv block. The block performs local feature extraction and global context modeling with channel-level concatenation similarly to the dense connectivity pattern in DenseNet, which maintains the lightweight property at the same time. Furthermore, we apply it for multiple layers of our proposed end-to-end network in short video classification task while the temporal dependency is captured via dilated convolutions and bidirectional GRU for better representation. Finally, our model outperforms both state-of-the-art convolutional models and self-attention models on three human action recognition datasets with considerably fewer parameters, which demonstrates the effectiveness.

Automatic Grammar Error Correction (GEC) detects and corrects various types of syntax, spelling, and grammatical errors. Different approaches such as rule-based, Statistical Machine Translation (SMT), and Neural Machine Translation (NMT) have been proposed. Among these approaches, NMT based on seq2seq multi-head attention (Transformer) performs the best. The key shortcoming of GEC seq2seq models with multiple encoder-decoder layers is that only the top layer is exploited in the subsequent processes. In addition, due to the exposure bias problem during inference, some of the previous target words are deleted and replaced by other words generated by the model itself, which leads to unsatisfactory output. This paper proposed GEC model based on seq2seq Transformer for low-resource languages such as Arabic to address these issues. Initially, we proposed a noising method for constructing synthetic parallel data to overcome the bottleneck arising from the lack of corpus. Furthermore, motivated by the success of capsule networks in computer vision, we used the Expectation-Maximization routing algorithm to dynamically aggregate information across layers in Arabic GEC. Moreover, to conquer the exposure bias problem, we introduced a bidirectional regularization term using Kullback-Leibler divergence in the training objective to improve the agreement between Right-to-left and Left-to-right models. Experiments performed on two benchmarks QALB-2014 and QALB-2015 showed that our proposed model achieved the best F1 score compared to the existing Arabic GEC systems.

Distributed Machine Learning (DML) at the edge has become an essential topic for providing low-latency intelligence near the data sources. However, both the development and testing of DMLs lack sufficient support. Reusable libraries that abstract the general functionalities of DMLs are needed for rapid development. Moreover, existing physical testbeds are usually small and lack network flexibility, while virtual testbeds like simulators and emulators lack fidelity. This paper proposes a novel hybrid testbed EdgeTB, which provides numerous emulated nodes to generate large-scale and network-flexible test environments while incorporating physical nodes to guarantee fidelity. EdgeTB manages physical nodes and emulated nodes uniformly and supports arbitrary network topologies between nodes through dynamic configurations. Importantly, we propose Role-oriented development to support the rapid development of DMLs. Through case studies and experiments, we demonstrate that EdgeTB provides convenience for efficiently developing and testing DMLs in various structures with high fidelity and scalability.

Computation partitioning is an important technique to improve the application performance by selectively offloading some computations from the mobile devices to the nearby edge cloud. In a dynamic environment in which the network bandwidth to the edge cloud may change frequently, the partitioning of the computation needs to be updated accordingly. The frequent updating of partitioning leads to high state migration cost between the mobile side and edge cloud. However, existing works don’t take the state migration overhead into consideration. Consequently, the partitioning decisions may cause significant network congestion and increase overall completion time tremendously. In this article, with considering the state migration overhead, we propose a set of novel algorithms to update the partitioning based on the changing network bandwidth. To the best of our knowledge, this is the first work on computation partitioning for stateful data stream applications in dynamic environments. The algorithms aim to alleviate the network congestion and minimize the make-span through selectively migrating state in dynamic edge cloud environments. Extensive simulations show our solution not only could selectively migrate state but also outperforms other classical benchmark algorithms in terms of make-span. The proposed model and algorithms will enrich the scheduling theory forstatefultasks, which has not been explored before.

Compared with audio data-based music genre classification, researches on symbolic data-based music are scarce. Existing methods generally utilize manually extracted features, which is very time-consuming and laborious, and use traditional classifiers for label prediction without considering specific music features. To tackle this issue, we propose the Melodic Phrase Attention Network (MPAN) for symbolic data-based music genre classification. Our model is trained in three steps: First, we adopt representation learning, instead of the traditional musical feature extraction method, to obtain a vectorized representation of the music pieces. Second, the music pieces are divided into several melodic phrases through melody segmentation. Finally, the Melodic Phrase Attention Network is designed according to music characteristics, to identify the reflection of each melodic phrase on the music genre, thereby generating more accurate predictions. Experimental results show that our proposed method is superior to baseline symbolic data-based music genre classification approaches, and has achieved significant performance improvements on two large datasets.

Dialogue policy learning based on reinforcement learning is difficult to be applied to real users to train dialogue agents from scratch because of the high cost. User simulators, which choose random user goals for the dialogue agent to train on, have been considered as an affordable substitute for real users. However, this random sampling method ignores the law of human learning, making the learned dialogue policy inefficient and unstable. We propose a novel framework, Automatic Curriculum Learning-based Deep Q-Network (ACL-DQN), which replaces the traditional random sampling method with a teacher policy model to realize the dialogue policy for automatic curriculum learning. The teacher model arranges a meaningful ordered curriculum and automatically adjusts it by monitoring the learning progress of the dialogue agent and the over-repetition penalty without any requirement of prior knowledge. The learning progress of the dialogue agent reflects the relationship between the dialogue agent's ability and the sampled goals' difficulty for sample efficiency. The over-repetition penalty guarantees the sampled diversity. Experiments show that the ACL-DQN significantly improves the effectiveness and stability of dialogue tasks with a statistically significant margin. Furthermore, the framework can be further improved by equipping with different curriculum schedules, which demonstrates that the framework has strong generalizability.

Deep reinforcement learning has shown great potential in training dialogue policies.However, its favorable performance comes at the cost of many rounds of interaction.Most of the existing dialogue policy methods rely on a single learning system, while the human brain has two specialized learning and memory systems, supporting to find good solutions without requiring copious examples.Inspired by the human brain, this paper proposes a novel complementary policy learning (CPL) framework, which exploits the complementary advantages of the episodic memory (EM) policy and the deep Q-network (DQN) policy to achieve fast and effective dialogue policy learning.In order to coordinate between the two policies, we proposed a confidence controller to control the complementary time according to their relative efficacy at different stages.Furthermore, memory connectivity and time pruning are proposed to guarantee the flexible and adaptive generalization of the EM policy in dialog tasks.Experimental results on three dialogue datasets show that our method significantly outperforms existing methods relying on a single learning system.

Although emotional conversation generation has attracted widespread attention in recent years, research works on the emotional interactional mechanism are still scarce, which makes it difficult for existing emotional dialogue system to automatically determine a suitable emotion type for conversation generation. Such a response emotion planning task is often difficult due to the “gap” problem: we need to predict the emotional probability distribution of the upcoming responses, which have not yet been generated. In this article, we propose a novel method, namely interactional emotion learning (IEL), which adopts an intuitive way to eliminate the “gap” problem: we design a variational learning network, called potential response learning, to learn the latent distribution of responses for given conversation in a semantic space. Then, we predict an appropriate emotion for response generation based on both the dialogue context and the learned latent distribution. Extensive experiments have been performed on three off-the-shelf conversation datasets, and the experimental results show that the proposed variational learning network significantly boosts the prediction ability of our approach, and our IEL method outperforms the state-of-the-art dialogue classification methods in the emotion planning task.

Mobile edge computation partitioning is an effective technique to improve the applications performance on the mobile devices by selectively offloading some computations from the devices to the nearby edge cloud. Most previous works focus on the computation partitioning for a single user. Recent works begin to study the computation partitioning in a multiple user environment in which a number of users compete for the constrained computation resources on the edge cloud. However, these works neglect the fact that the users normally also share the network resources to access the edge cloud, and thus the allocation of bandwidth to the users significantly affects the overall performance of the users. In this paper, we studynetwork aware mobile edge computation partitioning in multi-user environments, i.e., to decide for each user which parts of the application should be offloaded onto the edge cloud, and which others should be executed locally, and meanwhile to allocate the access bandwidth among the users, such that the average application performance of the users is maximized. This problem is new in that we consider the competition among users for the network bandwidth as well as the computation resources in a multi-user environment. With a set of novel models and formulations, we transform the problem into the classic Multi-class Multi-dimensional Knapsack Problem, and develop an effective algorithm, namely Performance Function Matrix based Heuristic (PFM-H), to solve it. We further consider the user mobility and design effective online algorithms that could be easily deployed in practical systems. Comprehensive trace driven simulations show that our proposed algorithm outperforms the benchmark algorithms significantly in the average application performance.

The proliferation of mobile devices and ubiquitous access of the wireless network enable many new mobile applications such as augmented reality, mobile gaming and so on. As the applications are latency sensitive, researchers propose to offload the complex computations of these applications to the nearby edge cloud, in order to reduce the latency. Existing works mostly consider the problem of partitioning the computations between the mobile device and the traditional cloud that has abundant resources. The proposed approaches can not be applied in the context of mobile edge cloud, because both the resources in the mobile edge cloud and the wireless access bandwidth to the edge cloud are constrained. In this paper, we studyjoint computation partitioning and resource allocation problemfor latency sensitive applications in mobile edge clouds. The problem is novel in that we combine the computation partitioning and the two-dimensional resource allocations in both the computation resources and the network bandwidth. We develop a new and efficient method, namely Multi-Dimensional Search and Adjust (MDSA), which is an offline algorithm, to solve the problem. We compare MDSA with the classic list scheduling method and theSearchAdjustalgorithm via comprehensive simulations. The results show that MDSA outperforms the benchmark algorithms in terms of the overall application latency. Moreover, we also design an online method, named by Cooperative Online Scheduling (COS), which can be easily deployed in practical systems. By extensive evaluations, we show that COS outperforms the benchmark methods by 25 percent on average.

Task-oriented dialogue systems provide a convenient interface to help users complete tasks. An important consideration for task-oriented dialogue systems is the ability to against the noise commonly existed in the real-world conversation. Both rule-based strategies and statistical modeling techniques can solve noise problems, but they are costly. In this paper, we propose a new approach, called Dynamic Reward-based Dueling Deep Dyna-Q (DR-D3Q). The DR-D3Q can learn policies in noise robustly, and it is easy to implement by combining dynamic reward and the Dueling Deep Q-Network (Dueling DQN) into Deep Dyna-Q (DDQ) framework. The Dueling DQN can mitigate the negative impact of noise on learning policies, but it is inapplicable to dialogue domain due to different reward mechanisms. Unlike typical dialogue reward function, we integrate dynamic reward that provides reward in real-time for agent to make Dueling DQN adapt to dialogue domain. For the purpose of supplementing the limited amount of real user experiences, we take the DDQ framework as the basic framework. Experiments using simulation and human evaluation show that the DR-D3Q significantly improve the performance of policy learning tasks in noisy environments.1

Data shuffling can improve the statistical performance of distributed machine learning. However, the obstruction of applying data shuffling is the high communication cost. Existing works use coding technology to reduce communication cost. These works assume a master-worker based storage architecture. However, due to the demand for unlimited storage on the master, the master-worker storage architecture is not always practical in common data centers. In this paper, we propose a new coding method for data shuffling in the decentralized storage architecture, which is built on a fat-tree based data center network. The method determines which data samples should be encoded together and from which the encoded package should be sent to minimize the communication cost. We develop a real-world test-bed to evaluate our method. The results show that our method can reduce the transmission time by 6.4% over the state-of-art coding method, and by 27.8% over Unicasting.

The proliferation of mobile devices and ubiquitous access of the wireless network enables many new mobile applications such as augmented reality, mobile gaming and so on. As the applications are latency sensitive, researchers propose to off load the complex computations of these applications to the nearby mobile edge cloud, in order to reduce the latency. Existing works mostly consider the problem of partitioning the computations between the mobile device and the traditional cloud that has abundant resources. The proposed approaches can not be applied in the context of mobile edge cloud, because both the resources in the mobile edge cloud and the wireless access bandwidth to the edge cloud are constrained. In this paper, we study joint computation partitioning and resource allocation problem for latency sensitive applications in mobile edge clouds. The problem is novel in that we combine the computation partitioning and the two-dimensional resource allocations in both the computation resources and the network bandwidth. We develop a new and efficient method, namely Multi-Dimensional Search and Adjust (MDSA), to solve the problem. We compares MDSA with the classic list scheduling method and the Search Adjust algorithm via comprehensive simulations. The results show that MDSA outperforms the benchmark algorithms in terms of the overall application latency.

Mobile edge cloud has been increasingly concerned by researchers due to its closer distance to mobile users than the traditional cloud on Internet. Offloading computations from mobile devices to the nearby edge cloud is an effective technique to accelerate the applications and/or save energy on the mobile devices. However, the mobile edge cloud usually has limited computation resources and constrained access bandwidth shared by multiple users in its proximity. Thus, allocation of resources and bandwidth among the users is significant to the overall application performance. In this paper, we study network aware multi-user computation partitioning problem in mobile edge clouds, i.e., to decide for each user which parts of the application should be offload onto the edge cloud, and which others should be executed locally, and meanwhile to allocate the access bandwidth among the users, such that the average application performance of the users is maximized. This problem is novel in that we consider the competition among users for both computing resources and bandwidth, and jointly optimizes the partitioning decisions with the allocation of resources and bandwidths among users, while most existing works either focus on the single user computation partitioning or study the multiple user computation partitioning without regard of the constrained network bandwidth. We first formulate the problem, and then transform it into the classic Multi-class Multi-dimensional Knapsack Problem and develop an effective algorithm, namely Performance Function Matrix based Heuristic (PFM-H), to solve it. Comprehensive simulations show that our proposed algorithm outperforms the benchmark algorithms significantly in the average application performance.