Automating the annotation of benthic imagery (i.e., images of the seafloor and its associated organisms, habitats, and geological features) is critical for monitoring rapidly changing ocean ecosystems. Deep learning approaches have succeeded in this purpose; however, consistent annotation remains challenging due to ambiguous seafloor images, potential inter-user annotation disagreements, and out-of-distribution samples. Marine scientists implementing deep learning models often obtain predictions based on one-hot representations trained using a cross-entropy loss objective with softmax normalization, resulting with a single set of model parameters. While efficient, this approach may lead to overconfident predictions for context-challenging datasets, raising reliability concerns that present risks for downstream tasks such as benthic habitat mapping and marine spatial planning. In this study, we investigated classification uncertainty as a tool to improve the labeling of benthic habitat imagery. We developed a framework for two challenging sub-datasets of the recently publicly available BenthicNet dataset using Bayesian neural networks, Monte Carlo dropout inference sampling, and a proposed single last-layer committee machine. This approach resulted with a > 95% reduction of network parameters to obtain per-sample uncertainties while obtaining near-identical performance compared to computationally more expensive strategies such as Bayesian neural networks, Monte Carlo dropout, and deep ensembles. The method proposed in this research provides a strategy for obtaining prioritized lists of uncertain samples for human-in-the-loop interventions to identify ambiguous, mislabeled, out-of-distribution, and/or difficult images for enhancing existing annotation tools for benthic mapping and other applications.

The hematology analytics used for detection and classification of small blood components is a significant challenge. In particular, when objects exists as small pixel-sized entities in a large context of similar objects. Deep learning approaches using supervised models with pre-trained weights (e.g., ImageNet), such as residual networks (ResNets) and vision transformers (ViTs) have demonstrated success for many applications. Unfortunately, when applied to images outside the domain of learned representations, these methods often result with less than acceptable performance. A strategy to overcome this can be achieved by using self-supervised models, where representations are learned from in-domain images and weights are then applied for downstream applications, such as semantic segmentation. Recently, masked autoencoders (MAEs) have proven to be effective in pre-training ViTs to obtain representations that captures global context information. By masking regions of an image and having the model learn to reconstruct both the masked and non-masked regions, the resulting weights can be used for various applications. However, if the sizes of the objects in images are less than the size of the mask (i.e., patch size), the global context information is lost, making it almost impossible to reconstruct the image. In this study, we investigated the effect of mask ratios and patch sizes for blood components using a "small-scale" MAE to obtain learned ViT encoder representations. We then applied the encoder weights to train a U-Net Transformer (UNETR) for semantic segmentation to obtain both local and global contextual information. Our experimental results demonstrates that both smaller mask ratios and patch sizes improve the reconstruction of images using a MAE. We also show the results of semantic segmentation with and without pre-trained weights, where smaller-sized blood components benefited with pre-training. Overall, our proposed method offers an efficient and effective strategy for the segmentation and classification of small objects.

The ongoing discussion of AGI is often mixed with claims about the ability of models to mimic brain functions. This position paper argues that mimicking human behavior at some level is not enough to understand brain functions. We point specifically to system-level organization of brain functions facilitated by complementary processes such as dual decision pathways, called System 1 and System 2 by Daniel Kahneman. System 1 resembles many of the abilities captured by deep learning, such as large language models (LLMs). System 2 is mostly associated with structural causal models (SCMs). We outline some important areas where more research is needed. This includes the interplay between multiple process pathways and how System 2 learns and evolves to represent the causal relations in our world during the lifetime of agents that are interacting with the world. We also argue about the danger of how systems beyond the capabilities of the human brain carry the risk of considerable harm to our society. Addressing them requires serious discussion.

In this work, we apply state-of-the-art self-supervised learning techniques on a large dataset of seafloor imagery, BenthicNet, and study their performance for a complex hierarchical multi-label (HML) classification downstream task. In particular, we demonstrate the capacity to conduct HML training in scenarios where there exist multiple levels of missing annotation information, an important scenario for handling heterogeneous real-world data collected by multiple research groups with differing data collection protocols. We find that, when using smaller one-hot image label datasets typical of local or regional scale benthic science projects, models pre-trained with self-supervision on a larger collection of in-domain benthic data outperform models pre-trained on ImageNet. In the HML setting, we find the model can attain a deeper and more precise classification if it is pre-trained with self-supervision on indomain data. We hope this work can establish a benchmark for future models in the field of automated underwater image annotation tasks and can guide work in other domains with hierarchical annotations of mixed resolution.1

The choice of activation functions and their motivation is a long-standing issue within the neural network community. Neuronal representations within artificial neural networks are commonly understood as logits, representing the log-odds score of presence of features within the stimulus. We derive logit-space operators equivalent to probabilistic Boolean logic-gates AND, OR, and XNOR for independent probabilities. Such theories are important to formalize more complex dendritic operations in real neurons, and these operations can be used as activation functions within a neural network, introducing probabilistic Boolean-logic as the core operation of the neural network. Since these functions involve taking multiple exponents and logarithms, they are computationally expensive and not well suited to be directly used within neural networks. Consequently, we construct efficient approximations named $\text{AND}_\text{AIL}$ (the AND operator Approximate for Independent Logits), $\text{OR}_\text{AIL}$, and $\text{XNOR}_\text{AIL}$, which utilize only comparison and addition operations, have well-behaved gradients, and can be deployed as activation functions in neural networks. Like MaxOut, $\text{AND}_\text{AIL}$ and $\text{OR}_\text{AIL}$ are generalizations of ReLU to two-dimensions. While our primary aim is to formalize dendritic computations within a logit-space probabilistic-Boolean framework, we deploy these new activation functions, both in isolation and in conjunction to demonstrate their effectiveness on a variety of tasks including tabular classification, image classification, transfer learning, abstract reasoning, and compositional zero-shot learning.

An ensemble of deep models is commonly used to provide more robust and accurate performance for medical image classification. A drawback of the most common ensemble aggregation operators is that they give the same importance to all models in the ensemble. As a consequence, they cannot identify weak models that may negatively influence the ensemble performance. In this work, we propose a new method based on the Dirichlet distribution and Mahalanobis distance to learn dynamic weights to an ensemble of deep learning models. Through this method, it is possible to reduce the influence of weak models for each new sample evaluated by the ensemble and perform online ensemble pruning. We evaluate this method for an ensemble of six well-known deep models applied to four medical imaging datasets. The experiments show that our method achieves the best balanced accuracy for 2 out of 4 datasets and increases the confidence of the ensemble predictions.

Lesion border detection is considered a crucial step in diagnosing skin cancer. However, performing such a task automatically is challenging due to the low contrast between the surrounding skin and lesion, ambiguous lesion borders, and the presence of artifacts such as hair. In this paper we propose a two-stage approach for skin lesion border detection: (i) segmenting the skin lesion dermoscopy image using U-Net, and (ii) extracting the edges from the segmented image using a novel approach we call FuzzEdge. The proposed approach is compared with another published skin lesion border detection approach, and the results show that our approach performs better in detecting the main borders of the lesion and is more robust to artifacts that might be present in the image. The approach is also compared with the manual border drawings of a dermatologist, resulting in an average Dice similarity of 87.7%.

Adding small, well crafted perturbations to the pixel values of input images leads to adversarial examples, so called because these perturbed images can drastically affect the accuracy of machine learning classifiers. Defenses against such attacks are being studied, often with varying results. In this study, we introduce a model called the Stochastic-Gated Partially Binarized Network (SGBN ), that incorporates binarization and input-dependent stochasticity. In particular, a gate module learns the probability that individual weights in corresponding convolutional filters should be masked (turned on or off). The gate module itself consists of a shallow convolutional neural network, and its sigmoid outputs are stochastically binarized and pointwise multiplied with corresponding filters in the convolutional layer of the main network. We test and compare our model with several related approaches, and to try to gain an understanding of our model, we visualize activations of some of the gating network outputs and their corresponding filters.

Internal models are important when agents make decisions based on predictions of future states and their utilities. However, using internal models for planning can be time consuming. Therefore, it can be useful to use a habitual system for repetitive tasks that can be executed faster and with reduced algorithmic resources. Current evidence suggests that the brain uses both control systems, planning and habitual systems for behavioural control, which then requires an arbitration between these two systems. In our previous work [1], we proposed an Arbitrated Predictive Actor-Critic (APAC), which is a neural architecture demonstrating cooperative mechanisms of planning and habitual control systems for one step mapping. The present study tests the ability of such a model to control a simulated two-joints robotic arm during multiple reaching tasks with movement limitations that require multiple steps to solve the task. Our results show that APAC can learn the multi-step learning under various conditions. Interestingly, the APAC tends to shift from planning to habits by taking actions predicted by a habitual controller over the training time.

Language modeling is a valuable component of generative natural language processing (NLP) tasks, and benefits from explicit representations of the inherent hierarchies in language. We investigate a commonly used architecture that captures the concept that words are built from characters, and modify the word encoding mechanism to use a feed forward neural network rather than a recurrent neural network (RNN). This feed forward architecture facilitates increased performance and a reduction in the number of parameters over models that use common RNN implementations. We investigate whether word representations benelit from position- invariant features in the characters, and lind that fixed-position representations are sufficient.

The aim of machine learning is to obtain a good model to correctly predict unseen data. In order to train such models, one needs a sufficient number of clean examples of the ground truth. However, some applications' datasets are not guaranteed to consist entirely of pure examples and might contain mislabeled data. Handling mislabeled data is a domain of outlier statistics and have been studied to some extent in the context of machine learning. Here we ask how does mislabeled data in a training set effect classification performance in deep neural networks. More specifically, motivated by an industrial application, we consider the case where the probability of the class mislabeling in the training set varies considerably between each class. We hence contrast in this paper the case of systematic mislabeling of one class to the more commonly studied situation of a uniform mislabeling between all classes. We demonstrate that the non-uniform mislabeling is more challenging than the more commonly studied uniform case. We also explicitly explore the dependence of our findings to the size of the training data which is not only a common limiting factor in industrial applications but which also has a large effect on the results. We demonstrate that deep networks have an inherent robustness when large datasets are available.

An emerging field in medical diagnostics is the study of micro-circulations in blood vessels. Several characteristics of the micro-circulations in blood vessels have been shown to predict inflammation in a patient's tissue. The characteristics are video recorded via a camera inserted into the subject. At present, the analysis of the videos are done manually by visual inspection to determine inflammation. In our paper, we propose a technique to automatically classify the videos as containing inflammation or not. Our technique uses a convolutional neural network which classifies many different segments of images from a video and averages the predictions. Our network achieves an accuracy of 83%. We further divide inflammation into extreme and moderate inflammation and our network achieves an accuracy of 80%. This is the first step in developing methods that can perform a better quantitative analysis of inflammation to speed up medical diagnosis.

Kohonen's self-organizing map (SOM) is used to map high-dimensional data into a low-dimensional representation (typically a 2-D or 3-D space) while preserving their topological characteristics. A major reason for its application is to be able to visualize data while preserving their relation in the high-dimensional input data space as much as possible. Here, we are seeking to go further by incorporating semantic meaning in the low-dimensional representation. In a conventional SOM, the semantic context of the data, such as class labels, does not have any influence on the formation of the map. As an abstraction of neural function, the SOM models bottom-up self-organization but not feedback modulation which is also ubiquitous in the brain. In this paper, we demonstrate a hierarchical neural network, which learns a topographical map that also reflects the semantic context of the data. Our method combines unsupervised, bottom-up topographical map formation with top-down supervised learning. We discuss the mathematical properties of the proposed hierarchical neural network and demonstrate its abilities with empirical experiments.

Biological organisms need to accurately infer which features of their environment predict future rewards and punishments for survival sake. This problem resembles linear regression, which finds parameter values expressing the linear relationship between features and an outcome. The least mean squares regression method generalizes well when there is little system noise and at least as many training data points (experiences) as input features. When this is not the case, feature selection may be applied to eliminate irrelevant features and improve generalization. Here, we show a biologically plausible approach to feature selection that computes the maximum likelihood estimate of Pearl's “Noisy OR” model. We show that this results in highlighting the features that are most correlated with the outcome at the expense of the least correlated. We extend this “relative correlation” approach to represent global inhibitory features and show that as additive noise and the number of irrelevant features are increased, relative correlation leads to substantially less prediction error on test data than does least means squares in a simple linear regression task. We demonstrate how relative correlation can be implemented in a dual pathway neural network and discuss some similarities between it and the basal ganglia.

In this paper, we propose a hierarchical neural network similar to the Radial Basis Function (RBF) Network. The proposed Restricted RBF (rRBF) executes a neighborhood-restricted activation function for its hidden neurons and consequently generates a unique topological map, which differs from the conventional Self-Organizing Map, in its internal layer. The primary objective of this study is to visualize and study the emergence of order in the structure and investigate the relation between the order and the learning performance of a hierarchical neural network.

Dynamic Neural Field models have been used extensively to model brain functions, but mostly through computer simulations. However, there are recent examples of applications in robotics that I will discuss in this presentation. I will also discuss neurocognitive robotics that has the aim of understanding brain functions in contrast to neuromorphic robotics that has mainly the aim of solving robotics tasks with biologically inspired method.

In this research we propose a trainable controller for a mobile robot based on a layered neural network, in which the hidden layer is a topographical map. In this study we focus not only on building a general controller that can be embedded to mobile robots running in physical environment, but also on building controllers with good internal plausibility. We consider that internal plausibility of the physical functionality acquired by robots during their learning phase is important in increasing their usability in real world tasks. The internal plausibility in this study can be obtained by associating the topographical map formed internally with the actions of the robots. Here, we run some experiments using a small robot, e-puck, and report the preliminary results of our study.

Models of self-organizing cortical maps have focused on demonstrations with single objects in the environment. Recently, the validity of a traditional biological model has been questioned for the case of multiple simultaneous input sources. Here we show that the standard model is able to self-organize with multiple inputs. However, we also show that the ability to self-organization can be enhanced considerably by including top-down attention as well as some noise. The model is also used to simulate the development of tuning curves.

In this research, we proposed a model of a hierarchical three-layered perceptron, in which the middle layer contains a two dimensional map where the topological relationship of the high dimensional input data (external world) are internally represented. The proposed model executes a two-phase learning algorithm where the supervised learning of the output layer is proceeded by a self-organization unsupervised learning of the hidden layer. The objective of this study is to build a simple neural network model which is more biologically realistic than the standard Multilayer Perceptron model and that can form an internal representation that supports its learning potential. The characteristics of the proposed model are demonstrated using several benchmark classification problems.

We fit a weight-dependent STDP rule to the classic data of Bi and Poo (1998), showing that this rule leads to slow learning in a simulation with an integrate-and-fire neuron. The slowness of learning is explained by an inequality between the range of initial weights in the data and the largest relative potentiation. We show that slow learning can be overcome with an increased learning rate, but that this approach leads to rapid forgetting in the presence of realistic levels of background spiking. Our study demonstrates that weight-dependent STDP rules, commonly used in neural simulations, have biologically unrealistic consequences. We discuss the implications of this finding for several interpretations of weight-dependent plasticity and STDP more generally, and recommend directions for further research.

A heuristic is proposed to address free parameter selection for Support Vector Machines, with the goals of improving generalization performance and providing greater insensitivity to training set selection. The many local extrema in these optimization problems make gradient descent algorithms impractical. The main point of the proposed heuristic is the inclusion of a model complexity measure to improve generalization performance. We also use simulated annealing to improve parameter search efficiency compared to an exhaustive grid search, and include an intensity-weighted centre of mass of the most optimum points to reduce volatility. We examine two standard classification problems for comparison, and apply the heuristic to bioinformatics and retinal electrophysiology classification.

Determining the most appropriate inputs to a model has a significant impact on the performance of the model and associated algorithms for classification, prediction, and data analysis. Previously, we proposed an algorithm ICAIVS which utilizes independent component analysis (ICA) as a preprocessing stage to overcome issues of dependencies between inputs, before the data being passed through to an input variable selection (IVS) stage. While we demonstrated previously with artificial data that ICA can prevent an overestimation of necessary input variables, we show here that mixing between input variables is common in real-world data sets so that ICA preprocessing is useful in practice. This experimental test is based on new measures introduced in this paper. Furthermore, we extend the implementation of our variable selection scheme to a statistical dependency test based on mutual information and test several algorithms on Gaussian and sub-Gaussian signals. Specifically, we propose a novel method of quantifying linear dependencies using ICA estimates of mixing matrices with a new linear mixing measure (LMM).

Proper initiation of saccadic eye movements depends on an intricate balance between exogenous and endogenous control mechanisms. The superior colliculus (SC) is a major site of signal integration that has been shown to drive the initiation of saccades in the brainstem. Previous work has shown that a winner-take-all mechanism implemented with a continuous attractor neural network (CANN) can explain and reproduce a multitude of behavioural findings, including the gap effect and the production of express saccades by K. Kopecz (1995) and by T.P. Trappenberg (2001). This investigation advances the CANN model of saccade initiation in several important ways in order to account for trial by trial adaptation of saccadic reaction times in a biologically plausible manner. A key hypothesis is that endogenous inputs to the intermediate layer of the SC can be adapted through motivationally-based feedback from other areas of the brain such as the basal ganglia or higher cortical areas.

Many spiking neuron models have been proposed over the last decades with varying computational complexity and abstraction from biological neurons. Among the few studies that have compared spiking models, little emphasis has been given to the formal description of calibration methods in tuning model parameters. We give an example of calibrating a leaky integrate-and-fire neuron with the first-spike time of a Hodgkin-Huxley neuron. We further demonstrate how model parameters can be tuned to minimize subthreshold differences in membrane potential. This example emphasizes the dependencies of calibration methods on other experimental parameters, complicating detailed comparisons of spiking models.

The biologically realistic model of selective visual attention by Deco et al. uses a continuous attractor neural network to simulate a saliency map in posterior parietal cortex. We test the ability of the model to explain experimental evidence on the distribution of spatial attention. The majority of evidence supports the view that attention is a unitary construct, but recent experiments provide evidence for split attention foci. We simulate two such experiments. Our results suggest that the ability to divide attention depends on sustained endogenous signals from short term memory to the saliency map, stressing the interplay between working memory mechanisms and attention. Our results also point to a mechanism whereby inhibitory endogenous signals may play a role in dividing attention, suggesting a possible mechanism for inhibition of return.

Inferior temporal cortex (IT) neurons have large receptive fields when a single effective object stimulus is shown against a blank background, but have much smaller receptive fields when the object is placed in a natural scene. Thus, translation invariant object recognition is reduced in natural scenes, and this may help object selection. We describe a model which accounts for this by competition within an attractor in which the neurons are tuned to different objects in the scene, and the fovea has a higher cortical magnification factor than the peripheral visual field. Further- more, we show that top-down object bias can increase the receptive field size, facilitating object search in complex visual scenes, and providing a model of object-based attention. The model leads to the prediction that introduction of a second object into a scene with blank background will reduce the receptive field size to values that depend on the closeness of the second object to the target stimulus. We suggest that mechanisms of this type enable the output of IT to be primarily about one object, so that the areas that receive from IT can select the object as a potential target for action.

The problem of input variable selection is well known in the task of modeling real-world data. In this paper, we propose a novel model-free algorithm for input variable selection using independent component analysis and higher order cross statistics. Experimental results are given which indicate that the method is capable of giving reliable performance and that it outperforms other approaches when the inputs are dependent.

We propose a scheme for pattern classifications in applications which include ambiguous data, that is, where pattern occupy overlapping areas in the feature space. Such situations frequently occur with noisy data and/or where some features are unknown. We demonstrate that it is advantageous to first detect those ambiguous areas with the help of training data and then to re-classify those data in these areas as ambiguous before making class predictions on test sets. This scheme is demonstrated with a simple example and benchmarked on two real world applications.

The problem of input variable selection is well known in the task of modeling real world data. In this paper we propose a novel model-free algorithm for input variable selection using independent component analysis and higher order cross statistics. Experimental results are given which indicate that the method is capable of giving reliable performance and that it outperforms other approaches when the inputs are dependent.

Most current technologies in modeling time varying processes aim to adapt a static model over time in what has become to be known as continuous learning. We propose here a different approach to the same problem domain that includes the time explicitly in the modeling. An example implementation of this strategy is given in a form of a multilayer perceptron with explicit time input. The performance of this approach is evaluating on a benchmark that was constructed to illustrate typical problems in industrial applications.