Abstract When the data distribution in a dataset is highly imbalanced or long-tailed, this can lead to biased deep network models that are ineffective at handling samples from the tail classes. The issue arises because the features of the samples in the head classes are dominant during the learning process. Supervised contrastive learning (SCL) and transfer of angular information (TAI) from head classes to tail classes are two methods proposed in recent literature. It was shown that, for a well-balanced dataset, SCL effectively improves class separation by pulling together samples from the same class while pushing away samples from different classes. However, SCL can remain biased towards the head classes since it does not directly address the effects of class imbalance during learning. TAI addresses challenges posed by long-tailed image classification but fails to achieve both intra-class compactness and inter-class separability. To address these limitations and exploit the strengths of both of these approaches, we propose a unique hybrid method that seamlessly integrates SCL and angular variance to mitigate the adverse effects of long-tailed data on deep learning models for image classification. The proposed method is named Supervised Angular Contrastive Loss with Balancing Classifier. Moreover, this paper finds that adding a suitably designed balancing technique can enhance results even further. Experiments on long-tailed datasets with different class imbalance ratios demonstrate that the proposed method significantly outperforms all existing baseline approaches.

Most Convolutional Neural Networks (CNNs) consist of a number of stages of decreasing spatial resolution and increasing channel dimension between succeeding stages, each stage is composed of convolutional blocks that are repeated a number of times. Previous research on very simple CNNs consisting of a number of convolutional layers in each stage demonstrated that the introduction of feedback loops around convolutional layers can improve results. This paper studies the effectiveness of recursion on convolutional blocks in a more general setting and aims at explaining the results. Four recent models, namely ResNet, Inception, MobileNet and DenseNet are considered in this study. It is found that for all but DenseNet, the recursive version produces results that are similar or better than their feedforward counterpart when the number of convolutional blocks are preserved. To understand this finding and to discriminate the functional behaviors of the feedforward and recursive counterparts, we embark on three investigations: (1) measuring the evolution of the contextualization of the neurons of the last layer using the effective receptive field concept; (2) comparing the position and the size of the global coverage of the networks using class activation maps; and (3) analyzing the evolution of the organization of the feature space prior to the classifier using the Silhouette score. The investigations reveal that the recursion of a convolutional block shares many similarities with the behavior of a sequence of that block, indicating that a recursive alternative consisting of a single physical layer, can be regarded as a “faithful simulation” of its deeper‘feedforward counterpart. We conclude that except for densely connected models, the recursion of convolutional blocks is a safe and powerful alternative enhancing modern network architectures.

The Recursive Convolutional Layer (RCL) is a module that wraps a recursive feedback loop around a convolutional layer (CL). The RCL has been proposed to address some of the shortcomings of Convolutional Neural Networks (CNNs), as its unfolding increases the depth of a network without increasing the number of weights. We investigated the “naïve” substitution of CL with RCL on three base models: a 4-CL model, ResNet, DenseNet and their RCL-ized versions: C-FRPN, R-ResNet, and R-DenseNet using five image classification datasets. We find that this one-to-one replacement significantly improves the performances of the 4-CL model, but not those of ResNet or DenseNet. This led us to investigate the implication of the RCL substitution on the 4-CL model which reveals, among a number of properties, that RCLs are particularly efficient in shallow CNNs. We proceeded to re-visit the first set of experiments by gradually transforming the 4-CL model and the C-FRPN into respectively ResNet and R-ResNet, and find that the performance improvement is largely driven by the training regime whereas any depth increase negatively impacts the RCL-ized version. We conclude that the replacement of CLs by RCLs shows great potential in designing high-performance shallow CNNs.

Freezing of Gait (FoG) is a common symptom of Parkinson's disease (PD), manifesting as a brief, episodic absence, or marked reduction in walking, despite a patient's intention to move. Clinical assessment of FoG events from manual observations by experts is both time-consuming and highly subjective. Therefore, machine learning-based FoG identification methods would be desirable. In this article, we address this task as a fine-grained human action recognition problem based on vision inputs. A novel deep learning architecture, namely, higher order polynomial transformer (HP-Transformer), is proposed to incorporate pose and appearance feature sequences to formulate fine-grained FoG patterns. In particular, a higher order self-attention mechanism is proposed based on higher order polynomials. To this end, linear, bilinear, and trilinear transformers are formulated in pursuit of discriminative fine-grained representations. These representations are treated as multiple streams and further fused by a cross-order fusion strategy for FoG detection. Comprehensive experiments on a large in-house dataset collected during clinical assessments demonstrate the effectiveness of the proposed method, and an area under the receiver operating characteristic (ROC) curve (AUC) of 0.92 is achieved for detecting FoG.

The development of Convolutional Neural Networks (CNNs) trends towards models with an ever growing number of Convolutional Layers (CLs) and increases the number of trainable parameters significantly. Such models are sensitive to these structural parameters, which implies that large models have to be carefully tuned using hyperparameter optimisation, a process that can be very time consuming. In this paper, we study the usage of Recursive Convolutional Layers (RCLs), a module relying on an algebraic feedback loop wrapped around a CL, which can replace any CL in CNNs. Using three publicly available datasets, CIFAR10, CIFAR100 and SVHN, and a simple model comprised of 4 RCLs, we compare its performances with those obtained by its feedforward counterpart, and exhibit some core properties and use-cases of RCLs. In particular, we show that RCLs can lead to models of better performances, and that reducing the number of modules from four to one lead to a decrease in accuracy of 3.5% on average for models using RCLs, compared to 23% using CLs. Hence, the resulting architecture is much more robust to the addition or the removal of layers. We conclude by relating the effects obtained using additional CLs with those obtained using additional recursion on RCLs, which provides incentives that the latter can simulate an increase of depth but with no extra cost of parameters. Such results point to the potential benefits of either selectively or replacing all CLs by RCLs, in most recently introduced CNNs.

The popularity of neural networks that explicitly utilise the global correlation structure of their features have become vastly more popular ever since the Transformer architecture was developed. We propose to embed unsupervised Self-Organising Maps within neural networks as a means to model the global correlation structure. By enforcing topological preservation therein, such a neural network is able to represent more complex correlation structures and to produce interpretable visualisations as a byproduct. We validate this approach by comparing with the existing state-of-the-art attention substitute within its own ‘Hamburger’ framework and by illustrating maps learnt by the module. Overall, this paper serves as a proof of concept for integrating Self-Organising Maps within a supervised network.

Freezing of gait (FoG) is identified as a sudden and brief episode of movement cessation despite the intention to continue walking. It is one of the most disabling symptoms of Parkinson's disease (PD) and often leads to falls and injuries. Many computer-aided FoG detection methods have been proposed to use data collected from unimodal sources, such as motion sensors, pressure sensors, and video cameras. However, there are limited efforts of multimodal-based methods to maximize the value of all the information collected from different modalities in clinical assessments and improve the FoG detection performance. Therefore, in this study, a novel end-to-end deep architecture, namely graph fusion neural network (GFN), is proposed for multimodal learning-based FoG detection by combining footstep pressure maps and video recordings. GFN constructs multimodal graphs by treating the encoded features of each modality as vertex-level inputs and measures their adjacency patterns to construct complementary FoG representations, thus reducing the representation redundancy among different modalities. In addition, since GFN is devised to process multimodal graphs of arbitrary structures, it is expected to achieve superior performance with inputs containing missing modalities, compared to the alternative unimodal methods. A multimodal FoG dataset was collected, which included clinical assessment videos and footstep pressure sequences of 340 trials from 20 PD patients. Our proposed GFN demonstrates a great promise of multimodal FoG detection with an area under the curve (AUC) of 0.882. To the best of our knowledge, this is one of the first studies to utilize multimodal learning for automated FoG detection, which offers significant opportunities for better patient assessments and clinical trials in the future.

Sign language translation (SLT), which generates text in a spoken language from visual content in a sign language, is important to assist the hard-of-hearing community for their communications. Inspired by neural machine translation (NMT), most existing SLT studies adopted a general sequence to sequence learning strategy. However, SLT is significantly different from general NMT tasks since sign languages convey messages through multiple visual-manual aspects. Therefore, in this paper, these unique characteristics of sign languages are formulated as hierarchical spatio-temporal graph representations, including high-level and fine-level graphs of which a vertex characterizes a specified body part and an edge represents their interactions. Particularly, high-level graphs represent the patterns in the regions such as hands and face, and fine-level graphs consider the joints of hands and landmarks of facial regions. To learn these graph patterns, a novel deep learning architecture, namely hierarchical spatio-temporal graph neural network (HST-GNN), is proposed. Graph convolutions and graph self-attentions with neighborhood context are proposed to characterize both the local and the global graph properties. Experimental results on benchmark datasets demonstrated the effectiveness of the proposed method.

SUMMARY Search services are the main interface through which people discover information on the Internet. A fundamental challenge in testing search services is the lack of oracles. The sheer volume of data on the Internet prohibits testers from verifying the results. Furthermore, it is difficult to objectively assess the ranking quality because different assessors can have very different opinions on the relevance of a Web page to a query. This paper presents a novel method for automatically testing search services without the need of a human oracle. The experimental findings reveal that some commonly used search engines, including Google, Yahoo!, and Live Search, are not as reliable as what most users would expect. For example, they may fail to find pages that exist in their own repositories, or rank pages in a way that is logically inconsistent. Suggestions are made for search service providers to improve their service quality. Copyright © 2010 John Wiley & Sons, Ltd.

The development of neural network (NN) models able to encode structured input, and the more recent definition of kernels for structures, makes it possible to directly apply machine learning approaches to generic structured data. However, the effectiveness of a kernel can depend on its sparsity with respect to a specific data set. In fact, the accuracy of a kernel method typically reduces as the kernel sparsity increases. The sparsity problem is particularly common in structured domains involving discrete variables which may take on many different values. In this paper, we explore this issue on two well-known kernels for trees, and propose to face it by recurring to self-organizing maps (SOMs) for structures. Specifically, we show that a suitable combination of the two approaches, obtained by defining a new class of kernels based on the activation map of a SOM for structures, can be effective in avoiding the sparsity problem and results in a system that can be significantly more accurate for categorization tasks on structured data. The effectiveness of the proposed approach is demonstrated experimentally on two relatively large corpora of XML formatted data and a data set of user sessions extracted from website logs.

Many underlying relationships among data in several areas of science and engineering, e.g., computer vision, molecular chemistry, molecular biology, pattern recognition, and data mining, can be represented in terms of graphs. In this paper, we propose a new neural network model, called graph neural network (GNN) model, that extends existing neural network methods for processing the data represented in graph domains. This GNN model, which can directly process most of the practically useful types of graphs, e.g., acyclic, cyclic, directed, and undirected, implements a function tau(G,n) is an element of IR(m) that maps a graph G and one of its nodes n into an m-dimensional Euclidean space. A supervised learning algorithm is derived to estimate the parameters of the proposed GNN model. The computational cost of the proposed algorithm is also considered. Some experimental results are shown to validate the proposed learning algorithm, and to demonstrate its generalization capabilities.

In this paper, we will consider the approximation properties of a recently introduced neural network model called graph neural network (GNN), which can be used to process-structured data inputs, e.g., acyclic graphs, cyclic graphs, and directed or undirected graphs. This class of neural networks implements a function tau(G,n) is an element of IR(m) that maps a graph G and one of its nodes n onto an m-dimensional Euclidean space. We characterize the functions that can be approximated by GNNs, in probability, up to any prescribed degree of precision. This set contains the maps that satisfy a property called preservation of the unfolding equivalence, and includes most of the practically useful functions on graphs; the only known exception is when the input graph contains particular patterns of symmetries when unfolding equivalence may not be preserved. The result can be considered an extension of the universal approximation property established for the classic feedforward neural networks (FNNs). Some experimental examples are used to show the computational capabilities of the proposed model.

The World Wide Web is an unregulated communication medium which exhibits very limited means of quality control. Quality assurance has become a key issue for many information retrieval services on the Internet, e.g. web search engines. This paper introduces some quality evaluation and assessment methods to assess the quality of Web pages. The proposed quality evaluation mechanisms are based on a set of quality criteria which were extracted from a targeted user survey. A weighted algorithmic interpretation of the most significant user quoted quality criteria is proposed. In addition, the paper utilizes machine learning methods to produce a prediction of quality for Web pages before they are downloaded. The set of quality criteria allows us to implement a Web search engine with quality ranking schemes, leading to Web crawlers which can crawl directly quality Web pages. The proposed approaches produce some very promising results on a sizeable Web repository.

The number of XML documents produced and available on the Internet is steadily increasing. It is thus important to devise automatic procedures to extract useful information from them with little or no intervention by a human operator. In this paper, we investigate the efficacy of an unsupervised learning approach, namely self-organising maps (SOMs), for the automatic clustering of XML documents. Specifically, we consider a relatively large corpus of XML formatted data from the INEX initiative and evaluate it using two different self-organising map models. The first model is the classical SOM model, and it requires the XML documents to be represented by real-valued vectors, obtained using a "bag of words" (or better a "bag of tags") approach. The other model is the SOM for structured data (SOM-SD) approach which is able to cluster structured data, and it is possible to feed the model with tree structured representations of the XML documents, thus explicitly preserving the structural information in the documents. The experimental results show that the SOM model exhibits quite a poor performance on this problem domain which requires the ability to encode structural properties of the data. The SOM-SD model, on the other hand, is able to produce a good clustering and generalization performance.

In this article, we present a new approach to page ranking. The page rank of a collection of Web pages can be represented in a parameterized model, and the user requirements can be represented by a set of constraints. For a particular parameterization, namely, a linear combination of the page ranks produced by different forcing functions, and user requirements represented by a set of linear constraints, the problem can be solved using a quadratic programming method. The solution to this problem produces a set of parameters which can be used for ranking all pages in the Web. We show that the method is suitable for building customized versions of PageRank which can be readily adapted to the needs of a vertical search engine or that of a single user.

An artificial neural network model, capable of processing general types of graph structured data, has recently been proposed. This paper applies the new model to the computation of customised page ranks problem in the World Wide Web. The class of customised page ranks that can be implemented in this way is very general and easy because the neural network model is learned by examples. Some preliminary experimental findings show that the model generalizes well over unseen Web pages, and hence, may be suitable for the task of page rank computation on a large Web graph.

The national health insurance system in Australia records details on medical services and claims provided to its population. An effective method to the discovery of temporal behavioral patterns in the data set is proposed in this paper. The method consists of a two-step approach which is applied recursively to the data set. First, a clustering algorithm is used to segment the data into classes. Then, hidden Markov models are employed to find the underlying temporal behavioral patterns. These steps are applied recursively to features extracted from the data set until convergence. The main objective is to minimize the misclassification of patient profiles into various classes. This results in a hierarchical tree model consisting of a number of classes; each class groups similar patient temporal behavioral patterns together. The capabilities of the proposed method are demonstrated through the application to a subset of the Australian national health insurance data set. It is shown that the proposed method not only clusters data into various categories of interest, but it also automatically marks the periods in which similar temporal behavioral patterns occurred.

This work proposes an improvement of a supervised learning technique for self organizing maps. The ideas presented in This work differ from Kohonen's approach to supervision in that a.) a rejection term is used, and b.) rejection affects the training only locally. This approach produces superior results because it does not affect network weights globally, and hence, prevents the addition of noise to the learning process of remote neurons. We implemented the ideas into self-organizing maps for structured data (SOM-SD) which is a more general form of self-organizing maps capable of processing graphs. The capabilities of the proposed ideas are demonstrated by utilizing a relatively large real world learning problem from the area of image recognition. It is shown that the proposed method produces better classification performances while being more robust and flexible than other supervised approaches to SOM.

In this paper, we consider the possibility of altering the PageRank of web pages, from an administrator's point of view, through the modification of the PageRank equation. It is shown that this problem can be solved using the traditional quadratic programming techniques. In addition, it is shown that the number of parameters can be reduced by clustering web pages together through simple clustering techniques. This problem can be formulated and solved using quadratic programming techniques. It is demonstrated experimentally on a relatively large web data set, viz., the WT10G, that it is possible to modify the PageRanks of the web pages through the proposed method using a set of linear constraints. It is also shown that the PageRank of other pages may be affected; and that the quality of the result depends on the clustering technique used. It is shown that our results compared well with those obtained by a HITS based method.

Recent developments in the area of neural networks produced models capable of dealing with structured data. Here, we propose the first fully unsupervised model, namely an extension of traditional self-organizing maps (SOMs), for the processing of labeled directed acyclic graphs (DAGs). The extension is obtained by using the unfolding procedure adopted in recurrent and recursive neural networks, with the replicated neurons in the unfolded network comprising of a full SOM. This approach enables the discovery of similarities among objects including vectors consisting of numerical data. The capabilities of the model are analyzed in detail by utilizing a relatively large data set taken from an artificial benchmark problem involving visual patterns encoded as labeled DAGs. The experimental results demonstrate clearly that the proposed model is capable of exploiting both information conveyed in the labels attached to each node of the input DAGs and information encoded in the DAG topology.

We introduce two new models which are obtained through the modification of the well known methods MLP and cascade correlation. These two methods differ fundamentally as they employ learning techniques and produce network architectures that are not directly comparable. We extended the MLP architecture, and reduced the constructive method to obtain very comparable network architectures. The greatest benefit of these new models is that we can obtain an MLP-structured network through a constructive method based on the cascade correlation algorithm, and that we can train a cascade correlation structured network using the standard MLP learning technique. Additionally, we show that cascade correlation is a universal approximator, a fact that has not yet been discussed in literature.