Pain assessment is a critical challenge in healthcare, requiring accurate and objective measurement to enhance patient care. Traditional methods rely on subjective self-reporting, which lacks reliability, particularly for patients with communication difficulties. This work presents PainFusion+, a multimodal transformer architecture for pain assessment that integrates physiological signals and facial expressions to improve pain assessment accuracy. For physiological signals, our approach first employs convolutional neural networks to extract local patterns from short signal fragments, capturing essential pain-related features. These localized representations are then processed by a transformer encoder, which models long-range dependencies to form a comprehensive global representation. For facial video data, we leverage a frozen video transformer to extract expressive features without requiring fine-tuning, significantly reducing computational costs. Finally, both feature spaces are fused using a transformer encoder, allowing effective cross-modal learning. Experiments on publicly available datasets demonstrate that PainFusion+ outperforms existing models. In biomedical signal processing, our method achieves over a 16 % improvement in accuracy. For multimodal pain estimation, it achieves 35.40 % accuracy on the BioVid dataset, setting a new state-of-the-art benchmark.

Computer vision is a branch of artificial intelligence that enables computers to learn patterns and extract useful information from various visual content. In healthcare, advances in computer vision models have been employed to provide accurate analysis of medical images, aiding tasks such as disease detection, human organ segmentation, and medical image classification. Such systems have the potential to enhance healthcare by improving patient outcomes and enabling early and accurate detection of diseases. The 3D CATBraTS is a novel deep learning architecture designed for three-dimensional segmentation of brain tumors on various magnetic resonance imaging (MRI) acquisitions. This paper presents a real-world web application capable of automatically detecting and accurately locating brain tumors on medical images. The proposed tool integrates the 3D CATBraTS as the backend deep learning model to improve its applicability and usability. The web application empowers medical professionals to perform segmentations seamlessly, even without coding skills, resulting in an improved user experience.

Brain tumour diagnosis is a challenging task yet crucial for planning treatments to stop or slow the growth of a tumour. In the last decade, there has been a dramatic increase in the use of convolutional neural networks (CNN) for their high performance in the automatic segmentation of tumours in medical images. More recently, Vision Transformer (ViT) has become a central focus of medical imaging for its robustness and efficiency when compared to CNNs. In this paper, we propose a novel 3D transformer named 3D CATBraTS for brain tumour semantic segmentation on magnetic resonance images (MRIs) based on the state-of-the-art Swin transformer with a modified CNN-encoder architecture using residual blocks and a channel attention module. The proposed approach is evaluated on the BraTS 2021 dataset and achieved quantitative measures of the mean Dice similarity coefficient (DSC) that surpasses the current state-of-the-art approaches in the validation phase.

Physical traits such as the shape of the hand and face can be used for human recognition and identification in video surveillance systems and in biometric authentication smart card systems, as well as in personal health care. However, the accuracy of such systems suffers from illumination changes, unpredictability, and variability in appearance (e.g. occluded faces or hands, cluttered backgrounds, etc.). This work evaluates different statistical and chrominance models in different environments with increasingly cluttered backgrounds where changes in lighting are common and with no occlusions applied, in order to get a reliable neural network reconstruction of faces and hands, without taking into account the structural and temporal kinematics of the hands. First a statistical model is used for skin colour segmentation to roughly locate hands and faces. Then a neural network is used to reconstruct in 3D the hands and faces. For the filtering and the reconstruction we have used the growing neural gas algorithm which can preserve the topology of an object without restarting the learning process. Experiments conducted on our own database but also on four benchmark databases (Stirling's, Alicante, Essex, and Stegmann's) and on deaf individuals from normal 2D videos are freely available on the BSL signbank dataset. Results demonstrate the validity of our system to solve problems of face and hand segmentation and reconstruction under different environmental conditions.

The contrast sensitivity function (CSF) characterizes spatial detection in the human visual system and is typically measured from simple, synthetic stimuli. We used spatial frequency decomposition, RMS contrast modulation, a yes/no paradigm and an adaptive staircase to measure isolated and contextual CSFs (iCSFs and cCSFs) from natural images. We employed Barten’s mechanistic model and adapted it for contextual modeling purposes by postulating that, signal detection in a given frequency band, when presented amongst other broadband signals, can be modeled as if amongst noise. We found that the iCSF varies with pictorial content, but that the standard CSF model and the image’s contrast spectrums are sufficient to predict with relative success the cCSF for any given image. We finally discuss the suitability of cCSF models in image quality modeling.

The International Joint Conference on Neural Networks (IJCNN) was held in Anchorage (Alaska) in May 2017. This top conference in the field of neural networks included many tracks and special sessions. In particular, a special session on Machine Learning Methods Neural Networks applied to Vision and Robotics (MLMVR) was organized by the authors receiving a large volume of excellent contributions. Only a small set of outstanding papers presented at this special session were invited to submit extended versions of their work. After a rigorous revision process, four of these papers were accepted.

This work presents the design of a real-time system to model visual objects with the use of self-organising networks. The architecture of the system addresses multiple computer vision tasks such as image segmentation, optimal parameter estimation and object representation. We first develop a framework for building non-rigid shapes using the growth mechanism of the self-organising maps, and then we define an optimal number of nodes without overfitting or underfitting the network based on the knowledge obtained from information-theoretic considerations. We present experimental results for hands and faces, and we quantitatively evaluate the matching capabilities of the proposed method with the topographic product. The proposed method is easily extensible to 3D objects, as it offers similar features for efficient mesh reconstruction.

Region merging algorithms commonly produce results that are seen to be far below the current commonly accepted state-of-the-art image segmentation techniques. The main challenging problem is the selection of an appropriate and computationally efficient method to control resolution and region homogeneity. In this paper we present a region merging algorithm that includes a semi-greedy criterion and an adaptive threshold to control segmentation resolution. In addition we present a new relative performance indicator that compares algorithm performance across many metrics against the results from human segmentation. Qualitative (visual) comparison demonstrates that our method produces results that outperform existing leading techniques.

Growing models have been widely used for clustering or topology learning. Traditionally these models work on stationary environments, grow incrementally and adapt their nodes to a given distribution based on global parameters. In this paper, we present an enhanced unsupervised self-organising network for the modelling of visual objects. We first develop a framework for building non-rigid shapes using the growth mechanism of the self-organising maps, and then we define an optimal number of nodes without overfitting or underfitting the network based on the knowledge obtained from information-theoretic considerations. This model is used to the representation of motion in image sequences by initialising a suitable segmentation. We present experimental results for hands and we quantitatively evaluate the matching capabilities of the proposed method with the topographic product.

Automatic sports video segmentation is a fast growth area of research in the visual information retrieval field. This paper presents a semi-parametric algorithm for parsing football video structures. The approach works on a two interleaved based process that closely collaborate towards a common goal. The core part of the proposed method focus performs a fast automatic football video annotation by looking at the enhance entropy variance within a series of shot frames. The entropy is extracted on the Hue parameter from the HSV color system, not as a global feature but in spatial domain to identify regions within a shot that will characterize a certain activity within the shot period. The second part of the algorithm works towards the identification of dominant color regions that could represent players and playfield for further activity recognition. Experimental results shows that the proposed football video segmentation algorithm performs with high accuracy.

We propose the design of a real-time system to recognize and interpret hand gestures. The acquisition devices are low cost 3D sensors. 3D hand pose segmentation, characterization and tracking will be implemented using the growing neural gas (GNG) structure. The capacity of the system to obtain information with a high degree of freedom allows the encoding of many gestures and a very accurate motion capture. The use of hand pose models combined with motion information provided with GNG permits to deal with the problem of the hand motion representation. A natural interface applied to a virtual mirror writing system and a module to estimate hand pose have been designed to demonstrate the validity of the system.

In our work we aim to explore a general framework that addresses the fundamental problem of universal unsupervised extraction of semantically meaningful visual regions. To this end this paper describes a novel region analysis technique using a self-organising map, the growing neural gas, which is adapted so as to improve modelling speed as well as to ensure a double-linkage chain around all region contours to simplify shape analysis. While the growing neural gas has been extensively applied to shape modelling, it has never explicitly been used for curvature analysis, contour description and region similarity. Once a contour network has been obtained, a transformation is applied that converts the closed contour to an open one, facilitating the use of certain angular descriptors. Discriminative descriptors derived from the properties of regions, their contours and their transformed contours are established and define a feature vector used for the representation of regions based on the appearance and contour information.

This paper aims to address the ability of self-organizing neural network models to manage real-time applications. Specifically, we introduce fAGNG (fast Autonomous Growing Neural Gas), a modified learning algorithm for the incremental model Growing Neural Gas (GNG) network. The Growing Neural Gas network with its attributes of growth, flexibility, rapid adaptation, and excellent quality of representation of the input space makes it a suitable model for real time applications. However, under time constraints GNG fails to produce the optimal topological map for any input data set. In contrast to existing algorithms the proposed fAGNG algorithm introduces multiple neurons per iteration. The number of neurons inserted and input data generated is controlled autonomous and dynamically based on a priory learnt model. Comparative experiments using topological preservation measures are carried out to demonstrate the effectiveness of the new algorithm to represent linear and non-linear input spaces under time restrictions.

The Self-Organising Artificial Neural Network Models, of which we have used the Growing Neural Gas (GNG) can be applied to preserve the topology of an input distribution. Traditionally these models neither do include local adaptation of the nodes nor colour information. In this paper, we present an extension to the original growing neural gas network that has probabilistic features and can be applied to preserve the topology of a non-stationary distribution. The network consists of the geometrical position of the nodes, the underline local feature structure of the image, and the distance vector between the modal image and any successive images. Accurate correspondence of the nodes between successive images, is measured through the calculation of the topographic product. The method performs continuously mapping over a distribution that changes over time and works with both smooth and abrupt changes. The method is successfully applied to object modelling and tracking.

The Self-Organising Artificial Neural Network Models, of which we have used the Growing Neural Gas (GNG) can be applied to preserve the topology of an input distribution. Traditionally these models neither do include local adaptation of the nodes nor colour information. In this paper, we extend GNG by presenting an improvement to the network that has both global and local properties and can track in cluttered backgrounds. The method performs continuously mapping over a distribution that changes over time and works with both smooth and abrupt changes. The central mechanism relies on the addition of global and local attributes, and skin colour information to the network which allow us to automatically model and track 2D gestures. Application to hand gesture video tracking is presented.

Self-organising neural networks have shown promise in a variety of applications areas. Their massive and intrinsic parallelism makes those networks suitable to solve hard problems in image-analysis and computer vision applications, especially when non-stationary environments occur. Moreover, this kind of neural networks preserves the topology of an input space by using their inherited competitive learning property. In this work we use a kind of self-organising network, the Growing Neural Gas, to solve some computer vision tasks applied to visual surveillance systems. It has been used their capacity to represent non rigid objects as a result of an adaptive process by a topology-preserving graph that constitutes an induced Delaunay triangulation of their shapes. The neural network is also modified to accelerate the learning algorithm in order to support applications with temporal constraints. This feature has been used to build a system able to track image features in video sequences. The system automatically keeps the correspondence of features among frames in the sequence using its own structure. Information obtained during the tracking process and allocated in the neural network can also be used to analyse the objects motion.

One of the tasks facing historians and conservationists is the authentication or dating of medieval manuscripts. To this end it is important to them to verify whether writings on the same or different manuscripts are concurrent. In this work we explore this task by capturing images of manuscript pages in infrared (IR) and modelling and then comparing the ink appearance of segmented text. The modelling of the text appearance relies on the unsupervised multimodal clustering of ink descriptors and the derived probability density functions. The similarity measure is built around the distribution of cluster labels and their proportions. We demonstrate our method by using both model inks of known composition and authentic Byzantine manuscripts.

In this work we explore the task of authenticating and dating ancient manuscripts by capturing images of pages in near infrared (NIR) and modelling and then comparing the ink appearance of segmented text. We present a texture feature descriptor to characterize and recognize semi-transparent materials such as the inks found in manuscripts. These textural patterns are different in nature from perceptual entities such as textons, tokens, frequency or repeatability of textural elements. Our ink texture descriptor relates a set of ink features from various first and second-order statistics to semi-liquid and viscous image-based properties of inks. In particular, we propose eigen features from the joint gray-level probabilities and off-diagonal sums of co-occurrence matrices. We test the qualities of the features with a classifier trained with the ink descriptor to show how well it recognizes eight different inks of known composition. Presented with the very same task the human visual system would fail to spot the ink composition difference given the inks inter-class and intra-class distances are extremely short.

This paper presents a robust approach to nonrigid modelling and tracking. The contour of the object is described by an active growing neural gas (A-GNG) network which allows the model to re-deform locally. The approach is novel in that the nodes of the network are described by their geometrical position, the underlying local feature structure of the image, and the distance vector between the modal image and any successive images. A second contribution is the correspondence of the nodes which is measured through the calculation of the topographic product, a topology preserving objective function which quantifies the neighbourhood preservation before and after the mapping. As a result, we can achieve the automatic modelling and tracking of objects without using any annotated training sets. Experimental results have shown the superiority of our proposed method over the original growing neural gas (GNG) network.

Inks found in Byzantine manuscripts are semi- transparent pigments and their examination and analysis provide an invaluable source of information on the authenticity and dating of manuscripts and the number of authors involved. However, inks are difficult to characterize because their intensity depends on the amount of liquid spread during scripting and the reflective properties of the support. Most existing methods for the analysis of ink materials are based on destructive testing techniques that require the physicochemical sampling of data. Such methods cannot be widely used because of the historical and cultural value of the manuscripts. In this work we show that manuscript inks can be represented through a mixture of Gaussian functions and can be characterised using co-occurrence matrices.

Modelling the appearance of 3D objects under very large pose variations relies on recovering correspondence between local features and the texture variations across views. However, changes in object 3D pose introduce self-occlusions and cause nonlinear variations in both the shape and the texture of object appearance. In this paper we introduce a pose-invariant appearance model that utilises a generic-view shape template for alignment, Kernel PCA for modelling shape and texture nonlinearities across views of large pose variations, and a neural network for model fitting to new images.

Recovering the shape of any 3D object using multiple 2D views requires establishing correspondence between feature points at different views. How-ever changes in viewpoint introduce self-occlusions, resulting nonlinear vari-ations in the shape and inconsistent 2D features between views. Here we introduce a multi-view nonlinear shape model utilising 2D view-dependent constraint without explicit reference to 3D structures. For nonlinear model transformation, we adopt Kernel PCA based on Support Vector Machines. 1

Recognition of human behaviours requires modeling the underlying spatial and temporal structures of their motion patterns. Such structures are intrinsically probabilistic and therefore should be modelled as stochastic processes. In this paper we introduce a framework to recognise behaviours based on both learning prior and continuous propagation of density models of behaviour patterns. Prior is learned from training sequences using hidden Markov models and density models are augmented by current visual observation.

An algorithm is described for modelling and recognising temporal structures of visual activities. The method is based on (1) learning prior probabilistic knowledge using hidden Markov models, (2) automatic temporal clustering of hidden Markov states based on expectation maximisation and (3) using observation augmented conditional density distributions to reduce the number of samples required for propagation and therefore improve recognition speed and robustness.