This paper presents a comprehensive empirical investigation into the interactions between various randomization techniques in Deep Neural Networks (DNNs) and their impact on learning performance. It is well-established that injecting randomness into the training process of DNNs, through various approaches, at different stages, is often beneficial for reducing overfitting and improving generalization. Nonetheless, the interactions between randomness techniques such as weight noise, dropout, and many others remain poorly understood. Consequently, it is challenging to determine which methods can be effectively combined to optimize DNN performance. To address this issue, we categorize the existing randomness techniques into four key types: injection of noise/randomness at the data, model structure, optimization or learning stage. We use this classification to identify gaps in the current coverage of potential mechanisms for the introduction of randomness, leading to proposing two new techniques: adding noise to the loss function and random masking of the gradient updates. In our empirical study, we employ a Particle Swarm Optimizer (PSO) for hyperparameter optimization (HPO) to explore the space of possible configurations to determine where and how much randomness should be injected to maximize DNN performance. We assess the impact of various types and levels of randomness for DNN architectures across standard computer vision benchmarks: MNIST, FASHION-MNIST, CIFAR10, and CIFAR100. Across more than 30 000 evaluated configurations, we perform a detailed examination of the interactions between randomness techniques and their combined impact on DNN performance. Our findings reveal that randomness through data augmentation and in weight initialization are the main contributors to performance improvement. Additionally, correlation analysis demonstrates that different optimizers, such as Adam and Gradient Descent with Momentum, prefer distinct types of randomization during the training process. A GitHub repository with the complete implementation and generated dataset is available1.

Evolutionary Algorithms (EAs) are often challenging to apply in real-world settings since evolutionary computations involve a large number of evaluations of a typically expensive fitness function. For example, an evaluation could involve training a new machine learning model. An approximation (also known as meta-model or a surrogate) of the true function can be used in such applications to alleviate the computation cost. In this paper, we propose a two-stage surrogate-assisted evolutionary approach to address the computational issues arising from using Genetic Algorithm (GA) for feature selection in a wrapper setting for large datasets. We define “Approximation Usefulness” to capture the necessary conditions to ensure correctness of the EA computations when an approximation is used. Based on this definition, we propose a procedure to construct a lightweight qualitative meta-model by the active selection of data instances. We then use a meta-model to carry out the feature selection task. We apply this procedure to the GA-based algorithm CHC (Cross generational elitist selection, Heterogeneous recombination and Cataclysmic mutation) to create a Qualitative approXimations variant, CHC Q X . We show that CHC Q X converges faster to feature subset solutions of significantly higher accuracy (as compared to CHC), particularly for large datasets with over 100K instances. We also demonstrate the applicability of the thinking behind our approach more broadly to Swarm Intelligence (SI), another branch of the Evolutionary Computation (EC) paradigm with results of PSO Q X , a qualitative approximation adaptation of the Particle Swarm Optimization (PSO) method. A GitHub repository with the complete implementation is available. 2

Vehicle utilization analysis is an essential tool for manufacturers to understand customer needs, improve equipment uptime, and to collect information for future vehicle and service development. Typically today, this behavioral modeling is done on high-resolution time-resolved data with features such as GPS position and fuel consumption. However, high-resolution data is costly to transfer and sensitive from a privacy perspective. Therefore, such data is typically only collected when the customer pays for extra services relying on that data. This motivated us to develop a multi-task ensemble approach to transfer knowledge from the high-resolution data and enable vehicle behavior prediction from low-resolution but high dimensional data that is aggregated over time in the vehicles. This study proposes a multi-task snapshot-stacked ensemble (MTSSE) deep neural network for vehicle behavior prediction by considering vehicles’ low-resolution operational life records. The multi-task ensemble approach utilizes the measurements to map the low-frequency vehicle usage to the vehicle behaviors defined from the high-resolution time-resolved data. Two data sources are integrated and used: high-resolution data called Dynafleet, and low-resolution so-called Logged Vehicle Data (LVD). The experimental results demonstrate the proposed approach’s effectiveness in predicting the vehicle behavior from low frequency data. With the suggested multi-task snapshot-stacked ensemble deep network, it is shown how low-resolution sensor data can highly contribute to predicting multiple vehicle behaviors simultaneously while using only one single training process.

Feature selection is an intractable problem, therefore practical algorithms often trade off the solution accuracy against the computation time. In this paper, we propose a novel multi-stage feature selection framework utilizing multiple levels of approximations, or surrogates. Such a framework allows for using wrapper approaches in a much more computationally efficient way, significantly increasing the quality of feature selection solutions achievable, especially on large datasets. We design and evaluate a Surrogate-Assisted Genetic Algorithm (SAGA) which utilizes this concept to guide the evolutionary search during the early phase of exploration. SAGA only switches to evaluating the original function at the final exploitation phase.We prove that the run-time upper bound of SAGA surrogate-assisted stage is at worse equal to the wrapper GA, and it scales better for induction algorithms of high order of complexity in number of instances. We demonstrate, using 14 datasets from the UCI ML repository, that in practice SAGA significantly reduces the computation time compared to a baseline wrapper Genetic Algorithm (GA), while converging to solutions of significantly higher accuracy. Our experiments show that SAGA can arrive at near-optimal solutions three times faster than a wrapper GA, on average. We also showcase the importance of evolution control approach designed to prevent surrogates from misleading the evolutionary search towards false optima.

Batteries are a safety-critical and the most expensive component for electric vehicles (EVs). To ensure the reliability of the EVs in operation, it is crucial to monitor the state of health of those batteries. Monitoring their deterioration is also relevant to the sustainability of the transport solutions, through creating an efficient strategy for utilizing the remaining capacity of the battery and its second life. Electric buses, similar to other EVs, come in many different variants, including different configurations and operating conditions. Developing new degradation models for each existing combination of settings can become challenging from different perspectives such as unavailability of failure data for novel settings, heterogeneity in data, low amount of data available for less popular configurations, and lack of sufficient engineering knowledge. Therefore, being able to automatically transfer a machine learning model to new settings is crucial. More concretely, the aim of this work is to extract features that are invariant across different settings. In this study, we propose an evolutionary method, called genetic algorithm for domain invariant features (GADIF), that selects a set of features to be used for training machine learning models, in such a way as to maximize the invariance across different settings. A Genetic Algorithm, with each chromosome being a binary vector signaling selection of features, is equipped with a specific fitness function encompassing both the task performance and domain shift. We contrast the performance, in migrating to unseen domains, of our method against a number of classical feature selection methods without any transfer learning mechanism. Moreover, in the experimental result section, we analyze how different features are selected under different settings. The results show that using invariant features leads to a better generalization of the machine learning models to an unseen domain.

Predicting energy (fuel and electric) consumption of hybrid vehicles is important on different levels: vehicle industry as a whole, individuals, and can also pave the way towards a more sustainable future. Despite its importance, providing accurate predictions is quite a challenging task. Many essential factors impacting energy consumption, including travel time, average speed, etc., needless to say, these features are not available beforehand. However, these factors are available in our data-set. To use these factors effectively, in this paper, we propose including them as different tasks in a multitask setting to help our main problem of energy consumption. The promise of this approach is that since these tasks are relevant, learning them together would provide a common feature space sharing information about all tasks. More importantly, this shared feature space would carry important information helping energy consumption in particular. In multitask learning, two important issues are task dominance and conflicting gradients of different tasks. Different studies have addressed these two separately. In this paper, we propose a method tackling these two problems simultaneously. We show experimentally the success of this method in comparison to state-of-the-art.

Ensemble learning methods combine multiple models to improve performance by exploiting their diversity. The success of these approaches relies heavily on the dissimilarity of the base models forming the ensemble. This diversity can be achieved in many ways, with well-known examples including bagging and boosting. It is the diversity of the models within an ensemble that allows the ensemble to correct the errors made by its members, and consequently leads to higher classification or regression performance. A mistake made by a base model can only be rectified if other members behave differently on that particular instance, and provide the aggregator with enough information to make an informed decision. On the contrary, lack of diversity not only lowers model performance, but also wastes computational resources. Nevertheless, in the current state of the art ensemble approaches, there is no guarantee on the level of diversity achieved, and no mechanism ensuring that each member will learn a different decision boundary from the others. In this paper, we propose a parallel orthogonal deep learning architecture in which diversity is enforced by design, through imposing an orthogonality constraint. Multiple deep neural networks are created, parallel to each other. At each parallel layer, the outputs of different base models are subject to Gram-Schmidt orthogonalization. We demonstrate that this approach leads to a high level of diversity from two perspectives. First, the models make different errors on different parts of feature space, and second, they exhibit different levels of uncertainty in their decisions. Experimental results confirm the benefits of the proposed method, compared to standard deep learning models and well-known ensemble methods, in terms of diversity and, as a result, classification performance.

Cloud computing offers a vast number of processing opportunities and heterogeneous resources and meets the requirements of numerous applications at various levels. Thus, the allocation and management of resources are vital in cloud computing. Resource allocation is a technique in which the available resources such as central processing unit, random-access memory, storage, and network bandwidth in cloud data centres are divided among users in a way that facilitates resource utilisation, provider profit, and user satisfaction. Integration and interaction with other modules of the resource management system, security, privacy, fairness, non-fragmentation of resources, resource utilisation, provider profit, user satisfaction, reducing energy consumption, load balancing, flexibility, scalability, availability, improvement the number and time of virtual machine migrations, and the number of overloaded resources are considered as challenges for the resource allocation mechanism. A systematic resource allocation survey with innovations in resource management system architecture, categorising mechanisms, addressing the challenges, and issues is presented. In addition to introducing the existing resource allocation mechanisms, other similar survey papers have been reviewed. Finally, there are some suggested topics for future work.

Creativity has a fundamental role in music composition. One of the theories, which exist about creativity, is combination-theory. In this paper the suitability of genetic algorithms and recurrent neural networks for modeling this theory is considered. We discuss that two phases of combination occurs: one at the genetic algorithm level, and the other at the network level. One important challenge in automatic composition is the musical fitness. We trained are current neural network on two pieces of music to learn the coarse and fine statistical regularities to define a fitness function. In our implementation, pitch and duration properties of notes are considered to generate new melodic passages. The paper concludes that this method can produce music with novel combinations of pre-existing ideas that sound faithful to the learned music.