The rising interest in Bayesian deep learning (BDL) has led to a plethora of methods for estimating the posterior distribution. However, efficient computation of inferences, such as predictions, has been largely overlooked with Monte Carlo integration remaining the standard. In this work we examine streamlining prediction in BDL through a single forward pass without sampling. For this, we use local linearisation of activation functions and local Gaussian approximations at linear layers. Thus allowing us to analytically compute an approximation of the posterior predictive distribution. We showcase our approach for both MLP and transformers, such as ViT and GPT-2, and assess its performance on regression and classification tasks. Open-source library: https://github.com/AaltoML/SUQ.

Deployment of deep neural networks in real-world settings typically requires adaptation to new tasks with few examples. Few-shot classification (FSC) provides a solution to this problem by leveraging pre-trained backbones for fast adaptation to new classes. However, approaches for multi-domain FSC typically result in complex pipelines aimed at information fusion and task-specific adaptation without consideration of the importance of backbone training. In this work, we introduce an effective strategy for backbone training and selection in multi-domain FSC by utilizing flatness-aware training and fine-tuning. Our work is theoretically grounded and empirically performs on par or better than state-of-the-art methods despite being simpler. Further, our results indicate that backbone training is crucial for good generalisation in FSC across different adaptation methods.

The receiver operating characteristic (ROC) is commonly used to analyze the performance of classifiers in data mining. An important topic in ROC analysis is the ROC convex hull (ROCCH), which is the least convex majorant (LCM) of the empirical ROC curve and covers potential optima for a given set of classifiers. ROCCH maximization problems have been taken as multiobjective optimization problem (MOPs) in some previous work. However, the special characteristics of ROCCH maximization problem makes it different from traditional MOPs. In this paper, the difference will be discussed in detail and a new convex hull-based multiobjective genetic programming (CH-MOGP) is proposed to solve ROCCH maximization problems. Specifically, convex hull-based without redundancy sorting (CWR-sorting) is introduced, which is an indicator-based selection scheme that aims to maximize the area under the convex hull. A novel selection procedure is also proposed based on the proposed sorting scheme. It is hypothesized that by using a tailored indicator-based selection, CH-MOGP becomes more efficient for ROC convex hull approximation than algorithms that compute all Pareto optimal points. Empirical studies are conducted to compare CH-MOGP to both existing machine learning approaches and multiobjective genetic programming (MOGP) methods with classical selection schemes. Experimental results show that CH-MOGP outperforms the other approaches significantly.

Evolution strategies (ESs) are powerful probabilistic search and optimization algorithms gleaned from biological evolution theory. They have been successfully applied to a wide range of real world applications. The modern ESs are mainly designed for solving continuous parameter optimization problems. Their ability to adapt the parameters of the multivariate normal distribution used for mutation during the optimization run makes them well suited for this domain. In this article we describe and study mixed integer evolution strategies (MIES), which are natural extensions of ES for mixed integer optimization problems. MIES can deal with parameter vectors consisting not only of continuous variables but also with nominal discrete and integer variables. Following the design principles of the canonical evolution strategies, they use specialized mutation operators tailored for the aforementioned mixed parameter classes. For each type of variable, the choice of mutation operators is governed by a natural metric for this variable type, maximal entropy, and symmetry considerations. All distributions used for mutation can be controlled in their shape by means of scaling parameters, allowing self-adaptation to be implemented. After introducing and motivating the conceptual design of the MIES, we study the optimality of the self-adaptation of step sizes and mutation rates on a generalized (weighted) sphere model. Moreover, we prove global convergence of the MIES on a very general class of problems. The remainder of the article is devoted to performance studies on artificial landscapes (barrier functions and mixed integer NK landscapes), and a case study in the optimization of medical image analysis systems. In addition, we show that with proper constraint handling techniques, MIES can also be applied to classical mixed integer nonlinear programming problems.

The design of software architecture is one of the difficult tasks in the modern component-based software development which is based on the idea that develop software systems by assembling appropriate off-the-shelf components with a well-defined software architecture. Component-based software development has achieved great success and been extensively applied to a large range of application domains from realtime embedded systems to online web-based applications. In contrast to traditional approaches, it requires software architects to address a large number of non-functional requirements that can be used to quantify the operation of system. Moreover, these quality attributes can be in conflict with each other. In practice, software designers try to come up with a set of different architectural designs and then identify good architectures among them. With the increasing scale of architecture, this process becomes time-consuming and error-prone. Consequently architects could easily end up with some suboptimal designs because of large and combinatorial search space. In this paper, we introduce AQOSA (Automated Quality-driven Optimization of Software Architecture) toolkit, which integrates modeling technologies, performance analysis techniques, and advanced evolutionary multiobjective optimization algorithms (i.e. NSGA-II, SPEA2, and SMS-EMOA) to improve non-functional properties of systems in an automated manner.

This paper discusses the adoption of self-adaptation for Evolutionary Algorithms operating in binary spaces using a direct encoding of the mutation rate. In particular, it focuses on the log-normal update rule for adapting the mutation rate, incorporated in a (mu, lambda)-strategy. Although it is well known that this update rule requires a lower boundary of the mutation rate to prevent it from collapsing to zero, the naive approach of enforcing a fixed lower boundary has undesirable side-effects. This paper studies the dynamics of the fixed lower boundary approach in depth and proposes a simple alternative for dealing with the lower boundary issue.

This paper discusses mixed integer evolution strategies (MIES) assisted by metamodels based on radial basis function networks (RBFN). The goal is to make MIES more suitable for optimization with time consuming evaluation functions.

In this paper we compare Mixed-Integer Evolution Strategies (MI-ES)and standard Evolution Strategies (ES)when applied to find optimal solutions for artificial test problems and medical image processing problems. MI-ES are special instantiations of standard ES that can solve optimization problems with different objective variable types (continuous, integer, and nominal discrete). Artificial test problems are generated with a mixed-integer test generator.The practical image processing problem iss the detection of the lumen boundary in IntraVascular UltraSound (IVUS)images. Based on the experimental results, it is shown that MI-ES generally perform better than standard ES on both artifical and practical image processing problems. Moreover it is shown that MI-ES can effectively improve the parameters settings for the IVUS lumen detection algorithm.