Multi-agent systems have recently received significant attention from researchers in many scientific fields. The value factorization method is popular for scaling up cooperative reinforcement learning in multi-agent environments. However, the approximation of the joint value function may introduce a significant disparity between the estimated and actual joint reward value function, leading to a local optimum for cooperative multi-agent deep reinforcement learning. In addition, as the number of agents increases, the input space grows exponentially, negatively impacting the convergence performance of multi-agent algorithms. This work proposes an efficient multi-agent rein-forcement learning algorithm, QDN, to enhance the convergence performance in cooperative multi-agent tasks. The proposed QDN scheme utilizes a competitive network to enable the agents to learn the value of the environmental state without the influence of actions. Hence, the error between the estimated joint reward value function and the actual joint reward value function can be significantly reduced, preventing the emergence of sub-optimal actions. Meanwhile, the proposed QDN algorithm utilizes the parametric noise on the network weights to introduce random-ness in the network's weights so that the agents can explore the environments and states effectively, thereby improving the convergence performance of the QDN algorithm. We evaluate the proposed QDN scheme using the SMAC challenges with various map difficulties. Experimental results show that the QDN algorithm excels in the convergence speed and the success rate in all scenarios compared to some state-of-the-art methods. Further experiments using four additional multi-agent tasks demonstrate that the QDN algorithm is robust in various multi-agent tasks and can significantly improve the training convergence performance compared with the state-of-the-art methods.

Wireless mesh networks (WMNs) have been deployed in many areas. There is an increasing demand for supporting a large number of mobile users in WMNs. As one of the key components in mobility management support, location management serves the purpose of tracking mobile users and locating them prior to establishing new communications. Previous dynamic location management schemes proposed for cellular and wireless local area networks (WLANs) cannot be directly applied to WMNs due to the existence of multihop wireless links in WMNs. Moreover, new design challenges arise when applying location management for silently roaming mobile users in the mesh backbone. Considering the number of wireless hops, an important factor affecting the performance of WMNs, we propose a DoMaIN framework that can help mobile users to decide whether an intra- or intergateway location update (LU) is needed to ensure the best location management performance (i.e., packet delivery) among dynamic location management solutions. In addition, by dynamically guiding mobile users to perform LU to a desirable location entity, the proposed DoMaIN framework can minimize the location management protocol overhead in terms of LU overhead in the mesh backbone. Furthermore, DoMaIN brings extra benefits for supporting a dynamic hop-based LU triggering method that is different from previous dynamic LU triggering schemes proposed for cellular networks and WLANs. We evaluate the performance of DoMaIN in different case studies using OPNET simulations. Comprehensive simulation results demonstrate that DoMaIN outperforms other location management schemes and is a satisfactory location management solution for a large number of mobile users silently and arbitrarily roaming under the wireless mesh backbone.

Wireless mesh networks (WMNs) have recently emerged to be a cost-effective solution to support large-scale wireless Internet access. Handoff management plays an important role in WMNs in delivering Quality of Service to mobile users. In this paper, the handoff support in Internet-based infrastructure WMNs is investigated. Previous solutions on handoff management in infrastructure WMNs mainly focus on intragateway mobility (e.g., single gateway is assumed in IEEE 802.11s WMNs). Furthermore, new handoff issues involved in intergateway mobility in WMNs (e.g., the network-layer handoff detection issue) have not been properly addressed. In this paper, a new architectural design, IMeX, is proposed to facilitate intergateway handoff management in Internet-based infrastructure WMNs. The proposed IMeX architecture is designed to specifically address the special handoff challenge in Internet-based WMNs. It can facilitate parallel executions of handoffs from multilayers, in conjunction with a data caching mechanism which guarantees minimum packet loss during handoffs. The required number and optimal placement of special mesh routers that form the IMeX architecture are modeled as a set covering problem which is solved based on a greedy algorithm. A comprehensive simulation study shows that the IMeX architecture enables fast handoffs and reestablishment of session communications in the intergateway mobility environment. Performance results demonstrate that the proposed IMeX architecture, with both the parallel handoff execution and data caching mechanism, outperforms the conventional WMN handoff schemes in terms of reducing handoff delays caused in multilayers and improving end-to-end performance of applications in WMNs.

Location management in wireless networks serves the purpose of tracking mobile nodes and locating them prior to establishing new communications. However, existing location management schemes in cellular and wireless local area networks cannot be directly applied to wireless mesh networks (WMNs) without non-trivial modifications due to the special properties of WMNs. In this paper, we propose a hybrid location tracking chain (Hatch) framework which includes a hierarchical architecture design and a hybrid location tracking scheme for location management in WMNs. Under the Hatch framework, a location update scheme is developed that takes the number of hops into consideration. Moreover, the proposed location tracking chain is formed by the routing protocol adopted by WMNs and the length of location tracking can be dynamically changed to improve the performance of packet delivery. We evaluate the proposed location management schemes in different case studies using OPNET simulations, the results of which show the efficiency of our proposed location management scheme in WMNs.

Location management in wireless networks serves the purpose of locating silently roaming mobile nodes (MNs) prior to establishing new communications. However, existing fixed location area (LA) designs and dynamic location management schemes proposed for cellular and wireless local area networks are not applicable to Internet-based infrastructure wireless mesh networks (IiWMNs) because they do not consider the special properties of IiWMNs. In this paper, we propose a resilient location area design (ReLoAD) for IiWMNs. Under ReLoAD, the formation of LAs can adapt to the changes of both paging load and service load of existing active data sessions in the mesh backbone network. Hence, the proposed ReLoAD can balance the tradeoff between signaling overhead caused by the paging procedure and MN power consumption caused by the location update procedure while maintaining the required Quality of Service (QoS) performance of existing traffic. We evaluate the proposed ReLoAD using OPNET simulations. Simulation results reveal that the proposed ReLoAD is a scalable solution for location management in IiWMNs.

Inter-domain handoffs in wireless mesh networks (WMNs) must evaluate the resource conditions of the target handoff domain in order to make QoS handoff decisions. Performance degradations to either or both handoff and existing QoS-aware traffic occur when resources on gateways and mesh routers in the handoff domain are over-utilized, while on the other hand, global resources across domains might still be underutilized. In this paper, we introduce a QoS-handoff framework for WMNs which includes 1) a resilient mesh architecture that offers dynamic gateway selection for inter-gateway handoffs under the assumption that different gateways belong to different domains and 2) a resilient forwarding scheme that allows intermediate mesh routers to make resilient next-hop decisions for traffic forwarding. With an integrated design and inter-dependency linkage of network architecture and traffic management, handoff resource utilization across domains can be realized. QoS requirements for both handoff and existing traffic can be achieved before and after handoffs in the inter-gateway WMN environment.

Handoff management plays an important role in wireless mesh networks (WMNs) in delivering Quality of Service to mobile users. Inter-gateway (across subnets) movement in WMNs usually requires the handoff support from multilayers and thus causes nonnegligible delays and packet loss. Previous solutions on handoff management in infrastructure WMNs mainly focus on intra-gateway mobility (e.g., single gateway is assumed in IEEE 802.11s WMNs) and exert the reduction of handoff delay so as to reduce packet loss. Furthermore, some handoff issues involved in inter-gateway mobility in WMNs (e.g., the network-layer handoff detection issue) have not been properly addressed. In this paper, we present a novel architectural design, namely Explicit multicast-based (Xcast-based) WMNs (XMesh), to facilitate inter-gateway handoff management. The proposed XMesh architecture enables parallel executions of handoffs from multilayers, in conjunction with a Xcast-based caching mechanism which builds on top of mesh routing protocols to guarantee minimum packet loss during handoffs in WMNs. The required number and optimal placement of special mesh routers that form the XMesh architecture are modeled as a set covering problem which is solved based on a greedy algorithm. A comprehensive simulation study shows that the XMesh architecture enables fast handoffs and re-establishment of session communications in the inter-gateway mobility environment. With both the parallel handoff execution and data caching mechanism, our architecture offers a seamless handoff for supporting real-time applications.

Wireless mesh networks (WMNs) have recently emerged to be a cost-effective solution to support large-scale wireless Internet access. One important component of realizing large-scale WMNs is mobility management. Although a considerable amount of research on handoff management for cellular and Mobile IP networks has been proposed, handoff management for IP-based WMNs remains largely unexplored. Conventional handoff mechanisms can cause significant performance degradation when directly applied to WMNs due to overlooking the key features of WMNs. They also do not address the special design challenges in WMNs. In this paper, a new architectural design is proposed to facilitate inter-gateway handoffs in WMNs. The proposed Planned Multicast Group (PMG)-based architecture can facilitate cross-layer handoffs and hence reduce the total handoff delay caused from multiple layers. Extensive simulations are conducted to evaluate the feasibility and efficiency of the proposed PMG approach. Simulation results demonstrate that the proposed PMG-based cross-layer handoff scheme outperforms the conventional WMN handoff schemes and is suitable for realtime multimedia applications in WMNs.