Recent deep learning approaches in single image super-resolution (SISR) can generate high-definition textures for super-resolved (SR) images. However, they tend to hallucinate fake textures and even produce artifacts. An alternative to SISR, reference-based SR (RefSR) approaches use high-resolution (HR) reference (Ref) images to provide HR details that are missing in the low-resolution (LR) input image. We propose a novel framework that leverages existing SISR approaches and enhances them with RefSR. Specifically, we refine the output of SISR methods using neural texture transfer, where HR features are queried from the Ref images. The query is conducted by computing the similarity of textural and semantic features between the input image and the Ref images. The most similar HR features, patch-wise, to the LR image is used to augment the SR image through an augmentation network. In the case of dissimilar Ref images from the LR input image, we prevent performance degradation by including the similarity scores in the input features of the network. Furthermore, we use random texture patches during the training to condition our augmentation network to not always trust the queried texture features. Different from past RefSR approaches, our method can use arbitrary Ref images and its lower-bound performance is based on the SR image. We showcase that our method drastically improves the performance of the base SISR approach.

Deep clustering algorithms combine representation learning and clustering by jointly optimizing a clustering loss and a non-clustering loss. In such methods, a deep neural network is used for representation learning together with a clustering network. Instead of following this framework to improve clustering performance, we propose a simpler approach of optimizing the entanglement of the learned latent code representation of an autoencoder. We define entanglement as how close pairs of points from the same class or structure are, relative to pairs of points from different classes or structures. To measure the entanglement of data points, we use the soft nearest neighbor loss, and expand it by introducing an annealing temperature factor. Using our proposed approach, the test clustering accuracy was 96.2% on the MNIST dataset, 85.6% on the Fashion-MNIST dataset, and 79.2% on the EMNIST Balanced dataset, outperforming our baseline models.

Poems are sequences of words that express ideas and emotions in an imaginative style, some following strict literary syntax or form. They are artistic expressions, and their generation requires in-depth knowledge and mastery of language. As such, poem generation is considered a very challenging task in Natural Language Processing and has been attracting research interest in the recent decade. We propose a method of generating poems using transformers, coupled with doc2vec embeddings in order to assess the automatically generated poems. In this method, we first train a transformer and a doc2vec model using a poem dataset. Then the trained transformer takes an input text and produces several generated poems. To have an objective basis for assessing the generated poems, we present a preliminary attempt at measuring the quality of a machine-generated poem by computing the cosine similarity score, referenced to the trained doc2vec model. This score is used as a basis for choosing the final output poem. The results show that this method ensures good cohesion between the machine-generated poem and the given input text. We then also explore the implication of the transformer training to the doc2vec embeddings of its output poems, which are shown to be more similar to poems (documents) in the train set as training progresses. Finally, we demonstrate how transformers can learn some poetry styles by exposing them to poems of specific poets.

Despite recent improvements, the arbitrary sizes of objects still impede the predictive ability of object detectors. Recent solutions combine feature maps of different receptive fields to detect multi-scale objects. However, these methods have large computational costs resulting to slower inference time, which is not practical for real-time applications. Contrarily, fusion methods depending on large networks with many skip connections demand larger memory requirement, prohibiting usage in devices with limited memory. In this paper, we propose a more computationally efficient fusion method which integrates higher-order information to low-level feature maps using a single operation. Our method can flexibly adapt to any base network, allowing tailored performance for different computational requirements. Our approach achieves 81.7% mAP at 41 FPS on the PASCAL VOC dataset using ResNet-50 as the base network, which is superior in terms of both speed and mAP as compared to several state-of-the-art baselines, even those which use larger base networks.

Advancements on text-to-image synthesis generate remarkable images from textual descriptions. However, these methods are designed to generate only one object with varying attributes. They face difficulties with complex descriptions having multiple arbitrary objects since it would require information on the placement and sizes of each object in the image. Recently, a method that infers object layouts from scene graphs has been proposed as a solution to this problem. However, their method uses only object labels in describing the layout, which fail to capture the appearance of some objects. Moreover, their model is biased towards generating rectangular shaped objects in the absence of ground-truth masks. In this paper, we propose an object encoding module to capture object features and use it as additional information to the image generation network. We also introduce a graph-cuts based segmentation method that can infer the masks of objects from bounding boxes to better model object shapes. Our method produces more discernible images with more realistic shapes as compared to the images generated by the current state-of-the-art method.

Resulting from technological advancements, it is now possible to regularly collect large volumes of data and to use these data for different applications. However, this obviously results in having very large numbers of samples as well as features. Dealing with high-volume and high-dimensional data is indeed a major challenge for machine learning algorithms, especially in terms of memory requirement and model training time. Fortunately, many of the features in the collected data are usually correlated, and some can be even be completely irrelevant for specific classification or pattern recognition tasks. By the nature of high-dimensional data, the large set of features can be reduced by removing redundant and irrelevant features. A two-stage feature selection algorithm based on feature redundancy and feature relevance is proposed in this paper. The proposed feature selection algorithm employs a hybrid model which combines filter and wrapper schemes to select the optimal feature subset. Five datasets from different domains are used to test the performance of the proposed feature selection algorithm based on three well-known machine learning algorithms, namely, k-Nearest Neighbor, Decision Trees, and Multilayer Perceptrons. Despite reducing the number of features using the proposed feature selection approach, the classification performances of the selected feature subsets are on par with or even significantly higher than the performance of the original feature set. Comparing with other state-of-the-art feature selection algorithms, the proposed method achieved higher classification accuracy with even lower number of features.

With the introduction of Convolutional Neural Networks, models for image classification achieve higher classification accuracy. Based on the pattern of the design of CNN architectures, increasing the number of layers equates to a higher classification accuracy, but also increases the number of parameters and model size. This negatively affects the model training time, processing time, and memory requirement. We develop ZipNet, a CNN architecture with a higher classification accuracy than ZFNet, the winner of ILSVRC 2013, but with 48.5× smaller model size and 48.7× fewer parameters. The classification accuracy of ZipNet is higher than the performance of ZFNet and SqueezeNet on all configurations of the Caltech-256 dataset with varying number of training examples.

Researches dealing with the task of style transfer have focused and produced results that only transfers the style of a reference image to the entirety of another image. As this domain would be beneficial in digital art, it would be preferable for such algorithms to support a more flexible style transfer, such that the style would be applied only on a specific portion of an image. We propose a framework that can selectively apply a given style onto an object using only 4 user-defined points. Our approach combines a style transfer module and an object segmentation module to synthesize the stylized image. As the ultimate goal is to develop an artistic application tool, we also introduce a method that makes use of specific filters to preserve certain characteristics of an image, such as its high frequency components. Experiments show that our proposed method is able to produce pleasing results with minimal effort while also being able to handle more complicated tasks such as the application of multiple reference style images onto different objects in an image.

Several time-critical problems relying on large amount of data, e.g., business trends, disaster response and disease outbreak, require cost-effective, timely and accurate data summary and visualization, in order to come up with an efficient and effective decision. Self-organizing map (SOM) is a very effective data clustering and visualization tool as it provides intuitive display of data in lower-dimensional space. However, with [Formula: see text] complexity, SOM becomes inappropriate for large datasets. In this paper, we propose a force-directed visualization method that emulates SOMs capability to display the data clusters with [Formula: see text] complexity. The main idea is to perform a force-directed fine-tuning of the 2D representation of data. To demonstrate the efficiency and the vast potential of the proposed method as a fast visualization tool, the methodology is used to do a 2D-projection of the MNIST handwritten digits dataset.

A structured 3D SOM is an extension of a Self-Organizing Map from 2D to 3D in such a way that a pre-defined structure is built into the design of the 3D map. The structured 3D SOM is a 3×3×3 structure that has a distinct core cube in the center and exterior cubes around the core. The current application of the structured SOM, as a digital music archive, only uses the 8 corner cubes among the 26 exterior cubes. Given that the SOM has a built-in structure, the SOM learning algorithm is modified to include a four-phase learning and labeling phase. The first phase is meant to position the music files in their general locations within the core cube. The second phase positions the music files in their respective corner cubes according to their music genre. The second phase is therefore a semi-supervised version of the SOM algorithm which leads to the stability of the trained SOM in terms of the general distribution of the music files in the core cube. The third phase does a fine adjustment of the weight vectors in the core cube and finalizes the training of the 3D SOM. The final fourth phase is the labeling of the core cube and the association (uploading) of music files to specific nodes in the core cube. Based on the pre-defined structure of the 3D SOM, a precise measure is developed to measure the quality of the resulting trained SOM (in this case, the music archive), as well as the quality of the different categories/genres of music albums based on a novel measure of the distortion values of music files with respect to their respective music genres.

With an electroencephalogram (EEG) sensor mounted on their head while learning mathematics using two computer-based learning software, EEG signals were collected from fifty six (56) academically-gifted students of ages 11 to 14. The EEG signals are used to predict four academic emotions, namely frustrated, confused, bored, and interested. It is shown that emotion classification accuracy is improved by selective prediction - performed only when a pre-determined proportion of EEG feature values deviate significantly from the baseline mean. The experiments on instances, where 0%, 2%, 4%, and up to 20% of the features are signifi- cantly stronger EEG signals, show that the accuracy rate of decision trees increases from 0.50, 0.59, and 0.45 (for instances with 0% special event features) to 0.74, 0.75, and 0.66 (for in- stances with 20% special event features) for predicting frustrated, confused and bored, respec- tively. Accuracy for predicting interested does not increase like for the other three emotions.

In this paper, we present an approach for sample selection using an ensemble of neural networks for credit scoring. The ensemble determines samples that can be considered outliers by checking the classification accuracy of the neural networks on the original training data samples. Those samples that are consistently misclassified by the neural networks in the ensemble are removed from the training dataset. The remaining data samples are then used to train and prune another neural network for rule extraction. Our experimental results on publicly available benchmark credit scoring datasets show that by eliminating the outliers, we obtain neural networks with higher predictive accuracy and simpler in structure compared to the networks that are trained with the original dataset. A rule extraction algorithm is applied to generate comprehensible rules from the neural networks. The extracted rules are more concise than the rules generated from networks that have been trained using the original datasets.

Keywords and key-phrases that concisely represent text documents are integral to many knowledge management and text information retrieval systems, as well as digital libraries in general. Not all text documents, however, are annotated with good keywords; and the quality of these keywords is often dependent on a tedious, sometimes manual, extraction and tagging process. To automatically extract high quality keywords without the need for a semantic analysis of the document, it is shown that artificial neural networks (ANN) can be trained to only consider in-document word features such as word frequency, word distribution in document, use of word in special parts of the document, and use of word formatting features (i.e. bold-faced, italicized, large-font size). Results show that purely local features are adequate in determining whether a word in a document is a keyword or not. Classification performance yields a G mean of a least 0.83, and weighted f-measure of 0.96 for both keywords and non-keywords. Precision for keywords alone, however, is not as high. To understand the basis for classifying keywords, C4.5 is used to extract rules from the ANN. The extracted rules from C4.5, in the form of a decision tree, show the relative importance of the different document features that were extracted.

The accuracy of neural networks can be improved when they are trained with discretized continuous attributes as additional inputs. Such input augmentation makes it easier for the network weights to form more accurate decision boundaries when the data samples of different classes in the data set are contained in distinct hyper-rectangular subregions in the original input space. In this paper we present first how a neural network can be trained with augmented discretized inputs. The additional inputs are obtained by simply dividing the original interval of each continuous attribute into subintervals of equal length. Thermometer encoding scheme is used to represent these discretized inputs. The network is then pruned to remove most of the discretized inputs as well as the original continuous attributes as long as the network still achieves a minimum preset accuracy requirement. We then discuss how MofN rules can be extracted from the pruned network by analyzing the activations of the network's hidden units and the weights of the network connections that remain in the pruned network. For data sets that have sample classes defined by relatively complex boundaries, surprisingly simple MofN rules with very good accuracy rates are obtained.

Keyword extraction is vital for Knowledge Management System, Information Retrieval System, and Digital Libraries as well as for general browsing of the web. Keywords are often the basis of document processing methods such as clustering and retrieval since processing all the words in the document can be slow. Common models for automating the process of keyword extraction are usually done by using several statistics-based methods such as Bayesian, K-Nearest Neighbor, and Expectation-Maximization. These models are limited by word-related features that can be used since adding more features will make the models more complex and difficult to comprehend. In this research, a Neural Network, specifically a backpropagation network, will be used in generalizing the relationship of the title and the content of articles in the archive by following word features other than TF-IDF, such as position of word in the sentence, paragraph, or in the entire document, and formats such as heading, and other attributes defined beforehand. In order to explain how the backpropagation network works, a rule extraction method will be used to extract symbolic data from the resulting backpropagation network. The rules extracted can then be transformed into decision trees performing almost as accurate as the network plus the benefit of being in an easily comprehensible format.

Even in an increasingly globalized market, the knowledge about individual local markets could still be invaluable. In this cross-national study of brand image perception of cars, survey data from buyers in the top 20 automobile markets have been collected, where each respondent has been asked to associate a car brand with individual brand images and corporate brand images. These data can be used as tool for decision making at the enterprise level. We describe an algorithm for constructing auto-associative neural networks which can be used as a tool for knowledge discovery from this data. It automatically determines the network topology by adding hidden units as they are needed to improve accuracy and by removing irrelevant input attributes. Two market research applications are presented, the first is for classification, and the second is for measuring similarities in the perceptions of the respondents from the different markets. In the first application, the constructed networks are shown to be more accurate than a decision tree. In the second application, the constructed networks are able to reproduce the training data very accurately and could be used to identify country-level (i.e. local) markets which share similar perceptions about the car brands being studied.

The MarB system is a digital archive of music files that are clustered and laid out as a self-organized map, following the SOM methodology for large digital archives. The system has the usual music archive features as follows: 1) automatic clustering and organization of music files into “islands of related music”; 2) classification of music clusters into various music genres; 3) playback of music files selected by the user; and 4) automatic generation of related music files for every music file that is chosen. In addition to these rather common features found in most Self-Organizing Maps (SOM) based digital music archives, MarB also allows for an interactive selection and clustering of sets and subsets of music files until a specific music file is found. This is done using a Scatter/Gather interface that allows the user to select interesting clusters of music files (gather mode), which are then re-organized and re-clustered (scatter mode) for the user to visually inspect and possibly listen to. The user is then asked to select new interesting clusters (gather mode again). This alternating selection and re-clustering process continues until the user chooses a specific music file, and is provided with a set of most related music files. A novel album dispersal measure is used to objectively assess the quality of the clusters produced both by the SOM and the special k -means algorithm employed in the Scatter-Gather module.

Learning in self-organizing maps (SOM) is considered unsupervised because training patterns do not need accompanying desired output information. Prior to its use in some real-world applications, however, a trained SOM often has to be labeled. This labeling phase is usually supervised in that labeled patterns need accompanying output information. Because such labeled patterns are not always available or may not even be possible to construct, the supervised nature of the labeling phase restricts the deployment of SOM from a wide range of potential domains of application. This work proposes a methodical and automatic SOM labeling procedure that does not require a set of prelabeled patterns. Instead, nodes in the trained map are clustered and subsets of training patterns associated to each of the clustered nodes are identified. Salient dimensions per node cluster that constitute the bases for labeling each node in the map are then identified. The effectiveness of the method is demonstrated on a SOM-based international market segmentation study.

Recent advances in algorithms that extract rules from artificial neural networks make it feasible to use neural networks as a tool for acquiring knowledge hidden in the data. Findings are reported from the use of such algorithms to separate core and noncore knowledge in a cross-national study of automobile brand image perception. Respondents from five Western European countries have been asked to associate individual and corporate brand associations for a number of well-known automobile brands. Knowledge, expressed as concise and accurate rules that distinguish between the respondents' perceptions of German and Japanese brands, is extracted from trained neural networks. This paper explains how both core knowledge, which captures the perceptions shared by the respondents in all countries, and country-specific noncore knowledge can be acquired and differentiated by a proposed two-step approach to train and extract rules from a multi-neural network system. The experimental results show that, in addition to providing a better understanding of the differences and similarities in the brand image perceptions of consumers in various countries, the proposed approach also yields better predictive accuracy than a decision tree method.

We investigate how knowledge acquired by a neural network from one input environment can be transferred and revised for similar application in a new environment. Knowledge revision is achieved by re-training the neural network. Knowledge common to both environments are retained, while localized knowledge components are introduced during network retraining. Various network performance measures are computed to measure how much knowledge is transferred and revised. Furthermore, because the knowledge acquired by a neural network can be expressed as an accurate set of simple rules, we are able to compare knowledge extracted from one network with that from another. In a cross-national study of car image perceptions, a comparison of the original and revised knowledge gives us insights into the commonalities and differences in brand perceptions across countries.

The WEBSOM methodology, proven effective for building very large text archives, includes a method that extracts labels for each document cluster assigned to nodes in the map. However, the WEBSOM method needs to retrieve all the words of all the documents associated to each node. Since maps may have more than 100,000 nodes and since the archive may contain up to seven million documents, the WEBSOM methodology needs a faster alternative method for keyword selection. Presented here is such an alternative method that is able to quickly deduce meaningful labels per node in the map. It does this just by analyzing the relative weight distribution of the SOM weight vectors and by taking advantage of some characteristics of the random projection method used in dimensionality reduction. The effectiveness of this technique is demonstrated on news document collections.

The Hopfield and bi-directional associative memory (BAM) models are well developed and carefully studied models for associative memory that are patterned after the memory structure of the animal brain. Their basic limitation is that they can only perform associations between at most two sets of patterns. Several different models for generalized associative memory are proposed. These models are all extensions of the Hopfield and BAM models that can perform multiple associations. Extensive software simulations are conducted to evaluate the different models, using memory capacity as the basis for comparing their performance. The use of the Widrow-Hoff gradient descent error correction algorithm is introduced that can improve the memory capacities of the various models. A potential application of these models as data fusion systems is explored.

Self-Organizing Maps, being used mainly with data that are not pre-labeled, need automatic procedures for extracting keywords as labels for each of the map units. The WEBSOM methodology for building very large text archives has a very slow method for extracting such unit labels. It computes the relative frequencies of all the words of all the documents associated to each unit and then compares these to the relative frequencies of all the words of all the other units of the map. Since maps may have more than 100,000 units and the archive may contain up to 7 million documents, the existing WEBSOM method is not practical. This paper describes how the meaningful labels per map unit can be deduced by analyzing the relative weight distribution of the SOM weight vectors and by taking advantage of some characteristics of the random projection method used in dimensionality reduction. The effectiveness of this technique is demonstrated on archives of the well studied Reuters and CNN collections. Comparisons with the WEBSOM method are provided.

Not only have self-organizing maps (SOMs), such as the WEBSOM, been shown to scale up to very large datasets, these maps also allow for a novel mode of navigating through a large collection of text documents. The entire text collection is presented to a user as a regular map, where each point in the map is associated to a group of documents that are likely to be composed of similar terms and phrases. In addition, the closer two points are in the map, the more similar are their respective associated documents. Thus, once an interesting document is found in the map, the user just has to click around the vicinity of that document to retrieve other similar documents. A major drawback of SOMs, however, is the long training time required, especially for document collections where both the volume and the dimensionality are huge. We demonstrate how the size of the initial text collection is progressively and drastically reduced from the raw document collection to the final SOM-based text archive. We demonstrate this using a widely studied Reuters collection.

The WEBSOM methodology for building very large text archives has a very slow method for extracting meaningful unit labels. This is because the method computes for the relative frequencies of all the words of all the documents associated to each unit and then compares these to the relative frequencies of all the words of all the other units of the map. Since maps may have more than 100,000 units and the archive may contain up to 7 million documents, the existing WEBSOM method is not practical. A fast alternative method is based on the distribution of weights in the weight vectors of the trained map, plus a simple manipulation of the random projection matrix used for input data compression. Comparisons made using a WEBSOM archive of the Reuters text collection reveal that a high percentage of keywords extracted using this method match the keywords extracted for the same map units using the original WEBSOM method.

We describe a method for multivariate function approximation which combines neural network learning, clustering and multiple regression. Neural networks with a single hidden layer are universal function approximators. However, due to the complexity of the network topology and the nonlinear transfer function used in computing the activation of the hidden units, the predictions of a trained network are difficult to comprehend. On the other hand, predictions from a multiple linear regression equation are easy to understand but not accurate when the underlying relationship between the input variables and the output variable as nonlinear. The method presented in this paper generates a set of multiple linear regression equations using neural networks. The number of regression equations as determined by clustering the weighted input variables. The predictions for samples in the same cluster are computed by the same regression equation. Experimental results on real-world data demonstrate that the new method generates relatively few regression equations from the training data samples. The errors an prediction using these equations are comparable to or lower than those achieved by existing function approximation methods.

Self-organizing maps (SOM) are proving to be useful as data analysis and visualization tools because they can visually render the data analysis results in 2D or 3D, and do not need category information for each input pattern. But this unsupervised nature of the training process makes it difficult to have separate training and test sets to determine the quality of the training process, which is done quite naturally for supervised neural network learning algorithms. In applications like data analysis, where there is little clue as to the way the SOM is supposed to look after training, it is important to be able to assess the quality of the self-organization process independent of the application, and without need for category information. The average unit disorder has been used to assess the quality of the ordering of a self-organized map. It is shown here that this same order metric can be used to assess the quality of the self-organization process itself. Based on this order metric, it can be determined whether the SOM has adequately learned, whether the parameters used to train the SOM have been correctly specified, and whether the SOM variant used is well-suited to the specific problem at hand.

This paper presents two metrics that measure the disorder of 2D self-organizing maps. These are the average direct neighbor distance and the average unit disorder. This theoretical work on the order of 2D self-organizing maps is done on Kristal maps, a variant of the original Kohonen model. It is shown that Kristal maps, when adequately trained, produce orderings that are superior to any of the known 2D orderings, such as the Canter-diagonal, Morton, Peano-Hilbert, raster-scan, row-prime, spiral, and random orderings.