We introduce a zero-shot merging framework for large language models (LLMs) that consolidates specialized domain experts into a single model without any further training.Our core contribution lies in leveraging relative task vectors-difference representations encoding each expert's unique traits with respect to a shared base model-to guide a principled and efficient merging process.By dissecting parameters into common dimensions (averaged across experts) and complementary dimensions (unique to each expert), we strike an optimal balance between generalization and specialization.We further devise a compression mechanism for the complementary parameters, retaining only principal components and scalar multipliers per expert, thereby minimizing overhead.A dynamic router then selects the most relevant domain at inference, ensuring that domain-specific precision is preserved.Experiments on code generation, mathematical reasoning, medical question answering, and instruction-following benchmarks confirm the versatility and effectiveness of our approach.Altogether, this framework enables truly adaptive and scalable LLMs that seamlessly integrate specialized knowledge for improved zero-shot performance.

Deep learning-based Natural Language Processing (NLP) models are vulnerable to adversarial attacks, where small perturbations can cause a model to misclassify.Adversarial Training (AT) is often used to increase model robustness.However, we have discovered an intriguing phenomenon: deliberately or accidentally miscalibrating models masks gradients in a way that interferes with adversarial attack search methods, giving rise to an apparent increase in robustness.We show that this observed gain in robustness is an illusion of robustness (IOR), and demonstrate how an adversary can perform various forms of test-time temperature calibration to nullify the aforementioned interference and allow the adversarial attack to find adversarial examples.Hence, we urge the NLP community to incorporate test-time temperature scaling into their robustness evaluations to ensure that any observed gains are genuine.Finally, we show how the temperature can be scaled during training to improve genuine robustness.* Equal contribution.This paper reports on the work VR did whilst at Amazon Web Services.VC holds concurrent appointments as an Amazon Scholar and as a faculty at EPFL.This paper describes the work performed at Amazon.

Training with mixed data distributions is a common and important part of creating multi-task and instruction-following models.The diversity of the data distributions and cost of joint training makes the optimization procedure extremely challenging.Data mixing methods partially address this problem, albeit having a sub-optimal performance across data sources and require multiple expensive training runs.In this paper, we propose a simple and efficient alternative for better optimization of the data sources by combining models individually trained on each data source with the base model using basic element-wise vector operations.The resulting model, namely Distribution Edited Model (DEM), is 11× cheaper than standard data mixing and outperforms strong baselines on a variety of benchmarks, yielding upto 6.2% improvement on MMLU, 11.5% on BBH, 16.1% on DROP, 6% on MathQA, and 9.3% on HELM with models of size 3B to 13B.Notably, DEM does not require full re-training when modifying a single data-source, thus making it very flexible and scalable for training with diverse data sources.The code is available at https://github.com/amazon-science/dem- distribution-edited-model.

In many real-world scenarios, data to train machine learning models becomes available over time. Unfortunately, these models struggle to continually learn new concepts without forgetting what has been learnt in the past. This phenomenon is known as catastrophic forgetting and it is difficult to prevent due to practical constraints. For instance, the amount of data that can be stored or the computational resources that can be used might be limited. Moreover, applications increasingly rely on large pre-trained neural networks, such as pre-trained Transformers, since compute or data might not be available in sufficiently large quantities to practitioners to train from scratch. In this paper, we devise a method to incrementally train a model on a sequence of tasks using pre-trained Transformers and extending them with Adapters. Different than the existing approaches, our method is able to scale to a large number of tasks without significant overhead and allows sharing information across tasks. On both image and text classification tasks, we empirically demonstrate that our method maintains a good predictive performance without retraining the model or increasing the number of model parameters over time. The resulting model is also significantly faster at inference time compared to Adapter-based state-of-the-art methods.

Creating open-ended algorithms, which generate their own never-ending stream of novel and appropriately challenging learning opportunities, could help to automate and accelerate progress in machine learning. A recent step in this direction is the Paired Open-Ended Trailblazer (POET), an algorithm that generates and solves its own challenges, and allows solutions to goal-switch between challenges to avoid local optima. However, the original POET was unable to demonstrate its full creative potential because of limitations of the algorithm itself and because of external issues including a limited problem space and lack of a universal progress measure. Importantly, both limitations pose impediments not only for POET, but for the pursuit of open-endedness in general. Here we introduce and empirically validate two new innovations to the original algorithm, as well as two external innovations designed to help elucidate its full potential. Together, these four advances enable the most open-ended algorithmic demonstration to date. The algorithmic innovations are (1) a domain-general measure of how meaningfully novel new challenges are, enabling the system to potentially create and solve interesting challenges endlessly, and (2) an efficient heuristic for determining when agents should goal-switch from one problem to another (helping open-ended search better scale). Outside the algorithm itself, to enable a more definitive demonstration of open-endedness, we introduce (3) a novel, more flexible way to encode environmental challenges, and (4) a generic measure of the extent to which a system continues to exhibit open-ended innovation. Enhanced POET produces a diverse range of sophisticated behaviors that solve a wide range of environmental challenges, many of which cannot be solved through other means.

This paper investigates the intriguing question of whether we can create learning algorithms that automatically generate training data, learning environments, and curricula in order to help AI agents rapidly learn. We show that such algorithms are possible via Generative Teaching Networks (GTNs), a general approach that is, in theory, applicable to supervised, unsupervised, and reinforcement learning, although our experiments only focus on the supervised case. GTNs are deep neural networks that generate data and/or training environments that a learner (e.g. a freshly initialized neural network) trains on for a few SGD steps before being tested on a target task. We then differentiate \emph{through the entire learning process} via meta-gradients to update the GTN parameters to improve performance on the target task. This paper introduces GTNs, discusses their potential, and showcases that they can substantially accelerate learning. We also demonstrate a practical and exciting application of GTNs: accelerating the evaluation of candidate architectures for neural architecture search (NAS). GTN-NAS improves the NAS state of the art, finding higher performing architectures when controlling for the search proposal mechanism. GTN-NAS also is competitive with the overall state of the art approaches, which achieve top performance while using orders of magnitude less computation than typical NAS methods. Speculating forward, GTNs may represent a first step toward the ambitious goal of algorithms that generate their own training data and, in doing so, open a variety of interesting new research questions and directions.

Reinforcement Learning agents with memory are constructed in this paper by extending neuroevolutionary algorithm NEAT to incorporate LSTM cells, i.e. special memory units with gating logic. Initial evaluation on POMDP tasks indicated that memory solutions obtained by evolving LSTMs outperform traditional RNNs. Scaling neuroevolution of LSTM to deep memory problems is challenging because: (1) the fitness landscape is deceptive, and (2) a large number of associated parameters need to be optimized. To overcome these challenges, a new secondary optimization objective is introduced that maximizes the information (Info-max) stored in the LSTM network. The network training is split into two phases. In the first phase (unsupervised phase), independent memory modules are evolved by optimizing for the info-max objective. In the second phase, the networks are trained by optimizing the task fitness. Results on two different memory tasks indicate that neuroevolution can discover powerful LSTM-based memory solution that outperform traditional RNNs.

Communication through vocalizations is used by spotted hyenas and chimpanzees for coordination during hunting and for raising alarm calls in defense (Bullinger et al., 2011; Holekamp et al., 2007). Vocal signals are omni-directional and are therefore more effective than visual communication in these situations. In cooperative tasks, agents use these signals to pro-actively exchange information for common good. A simulated predator-prey domain is considered in this paper- where multiple predator agents exchange real valued messages as an approximation of vocalization in nature. In artificial intelligence, the problem of coordination among multiple predator agents during prey capture is hard because of the non-Markovian environment (Panait and Luke, 2005). Experiments are carried out in this paper to show how information exchange through messaging can make the environment less non-Markovian and improve predator team performance during cooperative hunt. The values of these messages are analyzed to study the emergence of a common communication code among the predator agents. The results in this paper also provide an insight into the constraints under which language evolves in nature.

In nature there are teams of collaborators and competitors that evolve at the same time, yet computationally they have mostly been studied separately so far. This paper focuses on simultaneous cooperative and competitive coevolution in a complex predator-prey domain. Yong and Miikkulainen’s [3] Multi-Agent ESP architecture is extended to a Multi-Component ESP architecture consisting of multiple cooperating neural networks within an agent. This architecture successfully demonstrates hierarchical cooperation and competition in teams of prey and predators. In sustained coevolution in this complex domain, high-level pursuit-evasion behaviors emerge. In this manner, coevolution of neural networks is shown to scale up to an arms race of multiple competing and cooperating populations, more closely modeling coevolution in nature.