The rapid evolution of software libraries poses a considerable hurdle for code generation, necessitating continuous adaptation to frequent version updates while preserving backward compatibility. While existing code evolution benchmarks provide valuable insights, they typically lack execution-based evaluation for generating code compliant with specific library versions. To address this, we introduce GitChameleon 2.0, a novel, meticulously curated dataset comprising 328 Python code completion problems, each conditioned on specific library versions and accompanied by executable unit tests. GitChameleon 2.0 rigorously evaluates the capacity of contemporary large language models (LLMs), LLM-powered agents, code assistants, and RAG systems to perform version-conditioned code generation that demonstrates functional accuracy through execution. Our extensive evaluations indicate that state-of-the-art systems encounter significant challenges with this task; enterprise models achieving baseline success rates in the 48-51% range, underscoring the intricacy of the problem. By offering an execution-based benchmark emphasizing the dynamic nature of code libraries, GitChameleon 2.0 enables a clearer understanding of this challenge and helps guide the development of more adaptable and dependable AI code generation methods.

Transformer-based large language models (LLMs) have achieved remarkable success across natural language processing tasks, and their abilities have been exploited also for dynamic system control, particularly for robots. However, their practical deployment faces critical limitations: reliance on remote servers, substantial computational requirements, and high energy consumption make their practical application to real-time control very challenging. Small language models (SLMs), characterized by significantly fewer parameters, offer a promising alternative by enabling local execution, preserving privacy, and facilitating customization. Despite these advantages, SLMs remain underexplored for dynamic system control. This paper investigates whether SLMs can effectively control dynamic systems in real-time within a model-predictive control (MPC) framework. We propose a novel approach that leverages natural language prompting and a self-assessment strategy to generate effective cost functions for MPC, ensuring real-time adaptability and robustness. Unlike typical existing reward-shaping methods, our methodology is designed to be applicable to arbitrary systems and goals. Through extensive simulations, we test the methodology under diverse dynamic systems and tasks, validating the feasibility of SLM-based real-time control.

Large language model (LLM) agents for web interfaces have advanced rapidly, yet open-source systems still lag behind proprietary agents. Bridging this gap is key to enabling customizable, efficient, and privacy-preserving agents. Two challenges hinder progress: the reproducibility issues in RL and LLM agent training, where results often depend on sensitive factors like seeds and decoding parameters, and the focus of prior work on single-step tasks, overlooking the complexities of web-based, multi-step decision-making. We address these gaps by providing a statistically driven study of training LLM agents for web tasks. Our two-stage pipeline combines imitation learning from a Llama 3.3 70B teacher with on-policy fine-tuning via Group Relative Policy Optimization (GRPO) on a Llama 3.1 8B student. Through 240 configuration sweeps and rigorous bootstrapping, we chart the first compute allocation curve for open-source LLM web agents. Our findings show that dedicating one-third of compute to teacher traces and the rest to RL improves MiniWoB++ success by 6 points and closes 60\% of the gap to GPT-4o on WorkArena, while cutting GPU costs by 45\%. We introduce a principled hyperparameter sensitivity analysis, offering actionable guidelines for robust and cost-effective agent training.

We study the use of large language model-based agents for interacting with software via web browsers. Unlike prior work, we focus on measuring the agents’ ability to perform tasks that span the typical daily work of knowledge workers utilizing enterprise software systems. To this end, we propose WorkArena, a remote-hosted benchmark of 33 tasks based on the widely-used ServiceNow platform. We also introduce BrowserGym, an environment for the design and evaluation of such agents, offering a rich set of actions as well as multimodal observations. Our empirical evaluation reveals that while current agents show promise on WorkArena, there remains a considerable gap towards achieving full task automation. Notably, our analysis uncovers a significant performance disparity between open and closed-source LLMs, highlighting a critical area for future exploration and development in the field.

The ability of large language models (LLMs) to mimic human-like intelligence has led to a surge in LLM-based autonomous agents. Though recent LLMs seem capable of planning and reasoning given user instructions, their effectiveness in applying these capabilities for autonomous task solving remains underexplored. This is especially true in enterprise settings, where automated agents hold the promise of a high impact. To fill this gap, we propose WorkArena++, a novel benchmark consisting of 682 tasks corresponding to realistic workflows routinely performed by knowledge workers. WorkArena++ is designed to evaluate the planning, problem-solving, logical/arithmetic reasoning, retrieval, and contextual understanding abilities of web agents. Our empirical studies across state-of-the-art LLMs and vision-language models (VLMs), as well as human workers, reveal several challenges for such models to serve as useful assistants in the workplace. In addition to the benchmark, we provide a mechanism to effortlessly generate thousands of ground-truth observation/action traces, which can be used for fine-tuning existing models. Overall, we expect this work to serve as a useful resource to help the community progress towards capable autonomous agents. The benchmark can be found at https://github.com/ServiceNow/WorkArena.

A shared goal of several machine learning communities like continual learning, meta-learning and transfer learning, is to design algorithms and models that efficiently and robustly adapt to unseen tasks. An even more ambitious goal is to build models that never stop adapting, and that become increasingly more efficient through time by suitably transferring the accrued knowledge. Beyond the study of the actual learning algorithm and model architecture, there are several hurdles towards our quest to build such models, such as the choice of learning protocol, metric of success and data needed to validate research hypotheses. In this work, we introduce the Never-Ending VIsual-classification Stream (NEVIS'22), a benchmark consisting of a stream of over 100 visual classification tasks, sorted chronologically and extracted from papers sampled uniformly from computer vision proceedings spanning the last three decades. The resulting stream reflects what the research community thought was meaningful at any point in time, and it serves as an ideal test bed to assess how well models can adapt to new tasks, and do so better and more efficiently as time goes by. Despite being limited to classification, the resulting stream has a rich diversity of tasks from OCR, to texture analysis, scene recognition, and so forth. The diversity is also reflected in the wide range of dataset sizes, spanning over four orders of magnitude. Overall, NEVIS'22 poses an unprecedented challenge for current sequential learning approaches due to the scale and diversity of tasks, yet with a low entry barrier as it is limited to a single modality and well understood supervised learning problems. Moreover, we provide a reference implementation including strong baselines and an evaluation protocol to compare methods in terms of their trade-off between accuracy and compute. We hope that NEVIS'22 can be useful to researchers working on continual learning, meta-learning, AutoML and more generally sequential learning, and help these communities join forces towards more robust models that efficiently adapt to a never ending stream of data.

Explainability for machine learning models has gained considerable attention within the research community given the importance of deploying more reliable machine-learning systems. In computer vision applications, generative counterfactual methods indicate how to perturb a model’s input to change its prediction, providing details about the model’s decision-making. Current methods tend to generate trivial counterfactuals about a model’s decisions, as they often suggest to exaggerate or remove the presence of the attribute being classified. For the machine learning practitioner, these types of counterfactuals offer little value, since they provide no new information about undesired model or data biases. In this work, we identify the problem of trivial counterfactual generation and we propose DiVE to alleviate it. DiVE learns a perturbation in a disentangled latent space that is constrained using a diversity-enforcing loss to uncover multiple valuable explanations about the model’s prediction. Further, we introduce a mechanism to prevent the model from producing trivial explanations. Experiments on CelebA and Synbols demonstrate that our model improves the success rate of producing high-quality valuable explanations when compared to previous state-of-the-art methods. Code is available at https://github.com/ElementAI/beyond-trivial-explanations.

Modularity is a compelling solution to continual learning (CL), the problem of modeling sequences of related tasks. Learning and then composing modules to solve different tasks provides an abstraction to address the principal challenges of CL including catastrophic forgetting, backward and forward transfer across tasks, and sub-linear model growth. We introduce local module composition (LMC), an approach to modular CL where each module is provided a local structural component that estimates a module’s relevance to the input. Dynamic module composition is performed layer-wise based on local relevance scores. We demonstrate that agnosticity to task identities (IDs) arises from (local) structural learning that is module-specific as opposed to the task- and/or model-specific as in previous works, making LMC applicable to more CL settings compared to previous works. In addition, LMC also tracks statistics about the input distribution and adds new modules when outlier samples are detected. In the first set of experiments, LMC performs favorably compared to existing methods on the recent Continual Transfer-learning Benchmark without requiring task identities. In another study, we show that the locality of structural learning allows LMC to interpolate to related but unseen tasks (OOD), as well as to compose modular networks trained independently on different task sequences into a third modular network without any fine-tuning. Finally, in search for limitations of LMC we study it on more challenging sequences of 30 and 100 tasks, demonstrating that local module selection becomes much more challenging in presence of a large number of candidate modules. In this setting best performing LMC spawns much fewer modules compared to an oracle based baseline, however, it reaches a lower overall accuracy. The codebase is available under https://github.com/oleksost/LMC.

Finding neural network weights that generalize well from small datasets is difficult. A promising approach is to learn a weight initialization such that a small number of weight changes results in low generalization error. We show that this form of meta-learning can be improved by letting the learning algorithm decide which weights to change, i.e., by learning where to learn. We find that patterned sparsity emerges from this process, with the pattern of sparsity varying on a problem-by-problem basis. This selective sparsity results in better generalization and less interference in a range of few-shot and continual learning problems. Moreover, we find that sparse learning also emerges in a more expressive model where learning rates are meta-learned. Our results shed light on an ongoing debate on whether meta-learning can discover adaptable features and suggest that learning by sparse gradient descent is a powerful inductive bias for meta-learning systems.

We introduce and study the problem of Online Continual Compression, where one attempts to simultaneously learn to compress and store a representative dataset from a non i.i.d data stream, while only observing each sample once. A naive application of auto-encoder in this setting encounters a major challenge: representations derived from earlier encoder states must be usable by later decoder states. We show how to use discrete auto-encoders to effectively address this challenge and introduce Adaptive Quantization Modules (AQM) to control variation in the compression ability of the module at any given stage of learning. This enables selecting an appropriate compression for incoming samples, while taking into account overall memory constraints and current progress of the learned compression. Unlike previous methods, our approach does not require any pretraining, even on challenging datasets. We show that using AQM to replace standard episodic memory in continual learning settings leads to significant gains on continual learning benchmarks with images, LiDAR, and reinforcement learning agents.

Continual learning agents experience a stream of (related) tasks. The main challenge is that the agent must not forget previous tasks and also adapt to novel tasks in the stream. We are interested in the intersection of two recent continual-learning scenarios. In meta-continual learning, the model is pre-trained using meta-learning to minimize catastrophic forgetting of previous tasks. In continual-meta learning, the aim is to train agents for faster remembering of previous tasks through adaptation. In their original formulations, both methods have limitations. We stand on their shoulders to propose a more general scenario, OSAKA, where an agent must quickly solve new (out-of-distribution) tasks, while also requiring fast remembering. We show that current continual learning, meta-learning, meta-continual learning, and continual-meta learning techniques fail in this new scenario. We propose Continual-MAML, an online extension of the popular MAML algorithm as a strong baseline for this scenario. We show in an empirical study that Continual-MAML is better suited to the new scenario than the aforementioned methodologies including standard continual learning and meta-learning approaches.

Progress in the field of machine learning has been fueled by the introduction of benchmark datasets pushing the limits of existing algorithms. Enabling the design of datasets to test specific properties and failure modes of learning algorithms is thus a problem of high interest, as it has a direct impact on innovation in the field. In this sense, we introduce Synbols — Synthetic Symbols — a tool for rapidly generating new datasets with a rich composition of latent features rendered in low resolution images. Synbols leverages the large amount of symbols available in the Unicode standard and the wide range of artistic font provided by the open font community. Our tool's high-level interface provides a language for rapidly generating new distributions on the latent features, including various types of textures and occlusions. To showcase the versatility of Synbols, we use it to dissect the limitations and flaws in standard learning algorithms in various learning setups including supervised learning, active learning, out of distribution generalization, unsupervised representation learning, and object counting.

Continual learning, the setting where a learning agent is faced with a never-ending stream of data, continues to be a great challenge for modern machine learning systems. In particular the online or "single-pass through the data" setting has gained attention recently as a natural setting that is difficult to tackle. Methods based on replay, either generative or from a stored memory, have been shown to be effective approaches for continual learning, matching or exceeding the state of the art in a number of standard benchmarks. These approaches typically rely on randomly selecting samples from the replay memory or from a generative model, which is suboptimal. In this work, we consider a controlled sampling of memories for replay. We retrieve the samples which are most interfered, i.e. whose prediction will be most negatively impacted by the foreseen parameters update. We show a formulation for this sampling criterion in both the generative replay and the experience replay setting, producing consistent gains in performance and greatly reduced forgetting. We release an implementation of our method at https://github.com/optimass/MaximallyInterferedRetrieval

Underwater image mosaicking is an important tool for visual surveys, object detection, and as a means to control the underwater robot if done online. Such application areas can benefit significantly from a recent focus on robust methods for graph-based Simultaneous Localization and Mapping (SLAM). This paper focuses on two contributions: An approach to combine registration results from multiple methods in multimodal constraints and, up to the authors' knowledge, the first method to generate hyperedge constraints from state-of-the-art place recognition techniques. Both contributions are implemented within the Generalized Graph SLAM framework. Experimental results show that the methods generate informative constraints and that the authors' Prefilter method outperforms related methods on a large underwater image dataset processed with these methods.

This article discusses the possibility of building online a mosaic of the seafloor relying on a SLAM framework. The goal is to provide the ROV operator with an approximated seafloor's visual map relatively rough. In order to have that map, it is important to get an accurate estimate of the location of the visual landmarks and, in particular, a correct data association when a visual landmark is re-visited by the vehicle. The proposed approach uses the combination of a set of local mosaics constructed in the proximity of the SLAM visual landmarks instead of using a global mosaic. The algorithm was tested using a batch of experimental data in typical operating conditions and the results show the effectiveness of the approach.

Vessel maintenance entails periodic visual inspections of internal and external parts of the vessel hull in order to detect the typical defective situations affecting metallic structures, such as cracks, corrosion, etc. This paper presents the first steps towards a roboticized visual inspection system aiming at automating as much as possible the whole inspection process currently undertaken by human surveyors. Occurrence of defects are first outlined, followed by a brief description of the whole system. Next, details for one of the robotic platforms to be used are given, as well as for the image processing algorithms dealing with the visual detection of defects.

A line following guidance solution for underactuated marine systems is presented. The approach differs from other ones known in the literature in the definition of the error variables to be stabilized to zero. The proposed guidance technique has been applied to the Charlie USV (Unmanned Surface Vehicle), developed by CNR-ISSIA Autonomous robotic systems and control group, and experimental results are presented.

Determining a model of marine vehicles for the control purposes can be a very time-consuming process. Having this in mind, in [3] a much faster identification method has been proposed based on the use of self-oscillation identification method. Here we demonstrate the use of the proposed method for heading controller tuning of the autonomous unmanned surface vehicle (USV) Charlie.

The problem of low-cost high precision motion control of remotely operated vehicles (ROVs) in the proximity of the seabed through vision-based motion estimation is addressed in this paper. The proposed approach consists in the integration of a monocular vision system for the estimate of the vehicle’s linear motion with a dual-loop hierarchical architecture for kinematics and dynamics control. Results obtained by operating at sea the Romeo ROV are presented.

The design, development, sea trials and exploitation of the SESAMO (SEa Surface Autonomous MOdular unit) platform, an autonomous surface vessel for the study and characterization of the air-sea interface, are presented. The SESAMO prototype robot, developed by the robotics group of CNR-ISSIA, Genova branch, in strict cooperation with the scientific end-users in the framework of a project of the Italian National Program of Research in Antarctica (PNRA), was able to sample the sea surface microlayer and immediate sub-surface. After satisfactory basic at field trials of the navigation, guidance and control (NGC) and sampling systems, the robotised catamaran was exploited by the scientific end-users for water sample collection in the area of Terra Nova Bay, Ross Sea, Antarctica, in the framework of the PNRA project Chemical contamination.

An optical triangulation-correlation sensor for estimating the linear motion of underwater vehicles at low speed in the proximity of the seabed is described. Experimental results, obtained by post-processing image sequences acquired by the instrument mounted below the Romeo ROV in operating conditions, are presented.

This paper deals with a method to design the control architecture for mobile robots. A Petri net-based approach is extended in this work to deal with uncontrollable events. In particular, the case where an uncontrollable transition is present in the model of the task is tackled. This situation may arise, for example, when taking into account the initialization phase of the estimators.

The problem of designing an interface between the continuous-state and the discrete-state domains of intelligent control architectures is addressed, focusing on the case of a hierarchical navigation, guidance and control (NGC) architecture for unmanned underwater vehicles. The proposed interface represents the underlying continuous-state execution level as a discrete event system using a Petri net formalism. The correct behavior of the execution level is ensured, checking that no forbidden state is reached and that the proper task activation and deactivation order is respected. Methodologies for the off-line generation of the Petri net representation of the execution level and automatic system reconfiguration are presented.

This paper addresses the problem of guidance of unmanned underwater vehicles (UUVs). In the framework of a two layered hierarchical architecture decoupling the system dynamics and kinematics, two guidance laws for approaching a target with the desired orientation and following an environmental feature have been designed with Lyapunov-based techniques. Suitable acoustic-based estimators of the corresponding operational variables have been designed and integrated with the guidance and control system. Experimental results of pool trials of a prototype UUV executing free-space maneuvering and wall-following tasks are reported and discussed.

A two-layered hierarchical guidance and control architecture for UUVs has been integrated with a set of model-based motion estimators in order to enable the execution of high precision motion tasks in proximity of the seabed for scientific benthic applications. Preliminary tests, carried out in a high-diving pool, proved the system functionality, showing the high performances in terms of precision guaranteed by the use of PI-type guidance algorithms and combined estimation of the vehicle depth and altitude in hovering conditions.

This paper presents activities in the field of underwater robotics carried out at the Naval Automation Institute of the Italian National Research Council. The ultimate goal is the development of an autonomous underwater vehicle for research purposes. A bottom-up approach has been adopted and efforts are focused on the automation of undersea tasks. A prototype vehicle called ROBY has been designed and a dynamic AUV simulator, known as ROBYSIM has been implemented. Brief considerations on ROBY's technical characteristics and software control architecture are followed by a description of ROBYSIM. Work performed on the control and sensor filtering modules of the architecture is subsequently reviewed. Finally, experimental tests are briefly described and experimental results are compared to the outcome of simulation.