We consider the challenges of learning multi-agent/robot macro-action-based deep Q-nets including how to properly update each macro-action value and accurately maintain macro-action-observation trajectories. We address these challenges by first proposing two fundamental frameworks for learning macro-action-value function and joint macro-action-value function. Furthermore, we present two new approaches of learning decentralized macro-action-based policies, which involve a new double Q-update rule that facilitates the learning of decentralized Q-nets by using a centralized Q-net for action selection. Our approaches are evaluated both in simulation and on real robots.

In many real-world multi-robot tasks, high-quality solutions often require a team of robots to perform asynchronous actions under decentralized control. Decentralized multi-agent reinforcement learning methods have difficulty learning decentralized policies because of the environment appearing to be non-stationary due to other agents also learning at the same time. In this paper, we address this challenge by proposing a macro-action-based decentralized multi-agent double deep recurrent Q-net (MacDec-MADDRQN) which trains each decentralized Q-net using a centralized Q-net for action selection. A generalized version of MacDec-MADDRQN with two separate training environments, called Parallel-MacDec-MADDRQN, is also presented to leverage either centralized or decentralized exploration. The advantages and the practical nature of our methods are demonstrated by achieving near-centralized results in simulation and having real robots accomplish a warehouse tool delivery task in an efficient way.

BACKGROUND: Age-related macular degeneration (AMD) is the leading cause of irreversible visual loss in the elderly in developed countries and typically affects more than 10% of individuals over age 80. AMD has a large genetic component, with heritability estimated to be between 45% and 70%. Numerous variants have been identified and implicate various molecular mechanisms and pathways for AMD pathogenesis but those variants only explain a portion of AMD's heritability. The goal of our study was to estimate the cumulative genetic contribution of common variants on AMD risk for multiple pathways related to the etiology of AMD, including angiogenesis, antioxidant activity, apoptotic signaling, complement activation, inflammatory response, response to nicotine, oxidative phosphorylation, and the tricarboxylic acid cycle. While these mechanisms have been associated with AMD in literature, the overall extent of the contribution to AMD risk for each is unknown. METHODS: In a case-control dataset with 1,813 individuals genotyped for over 600,000 SNPs we used Genome-wide Complex Trait Analysis (GCTA) to estimate the proportion of AMD risk explained by SNPs in genes associated with each pathway. SNPs within a 50 kb region flanking each gene were also assessed, as well as more distant, putatively regulatory SNPs, based on DNaseI hypersensitivity data from ocular tissue in the ENCODE project. RESULTS: We found that 19 previously associated AMD risk SNPs contributed to 13.3% of the risk for AMD in our dataset, while the remaining genotyped SNPs contributed to 36.7% of AMD risk. Adjusting for the 19 risk SNPs, the complement activation and inflammatory response pathways still explained a statistically significant proportion of additional risk for AMD (9.8% and 17.9%, respectively), with other pathways showing no significant effects (0.3% - 4.4%). DISCUSSION: Our results show that SNPs associated with complement activation and inflammation significantly contribute to AMD risk, separately from the risk explained by the 19 known risk SNPs. We found that SNPs within 50 kb regions flanking genes explained additional risk beyond genic SNPs, suggesting a potential regulatory role, but that more distant SNPs explained less than 0.5% additional risk for each pathway. CONCLUSIONS: From these analyses we find that the impact of complement SNPs on risk for AMD extends beyond the established genome-wide significant SNPs.