The computational modeling of multi-sensory perception is challenging. It is important to understand and model temporal phenomena such as synchronization and synaptic integration over varying time intervals. Previously, we developed a sparse spatio-temporal model using networks of interconnected oscillators. This model achieved a binding of related object features in multiple sensory pathways.We apply this model to address a controversy in the field of psychology and education concerning the theory of learning styles. According to a popularly held view, learners fall into categories such as visual learners or auditory learners, and prefer to imbibe materials that match their learning styles. This view has been disputed as this categorization does not result in a demonstrable improvement in learning.We model preferences for a learning style through variations in synaptic learning efficacies. Hence a visual learner will possess higher synaptic efficacy in the visual pathway. The results of our simulation with the sparse spatio-temporal model show that there is little benefit to having or exploiting such differences in synaptic efficacy. The best system performance is achieved when all available sensory pathways are utilized rather than favoring a preferred pathway.Our model weighs in on this controversy, and supports the side that is against the categorization of students based on their learning styles. Since the idea of learning styles is quite prevalent in the literature and used in schools, it is important to disseminate appropriate research findings including those that utilize a computational neuroscience perspective, as in the current paper.

In our earlier work, we presented BOAT, a big-data open source analytics toolkit and framework, and applied it to analyze trends and outliers in public healthcare data. In this paper, we extend this framework to predict the costs of different medical procedures that patients could incur, based on open healthcare data. Specifically, we analyze de-identified patient data from New York State SPARCS (statewide planning and research cooperative system), consisting of more than 2 million records. We investigated the three model classes consisting of multiple linear regression, regression trees and deep neural networks (DNNs). We conducted a grid-search to identify the best parameter choices. We determined that the best performance based on grid-search cross validation with the widely used R2metric was achieved by an 8 layered DNN with size 5x5x10x25x25x10x5x5 using an Adam optimizer with learning rate of 0.01. We obtained an R2value of 0.71 which is better than the values reported in the literature for similar problems.

The interactive exploration of large and evolving datasets is challenging as relationships between underlying variables may not be fully understood. There may be hidden trends and patterns in the data that are worthy of further exploration and analysis. We present a system that methodically explores multiple combinations of variables using a searchlight subsect scan algorithm to identify outliers. An iterative k-means clustering algorithm is applied to features derived through a split-apply-combine paradigm used in the database literature. Outliers are identified as singleton or small clusters. This algorithm is swept across the dataset in a searchlight manner. The dimensions that contain outliers are combined with other dimensions using an efficient subset scan technique to gain further insight into the outliers. We illustrate this system by analyzing open health care data released by New York State. We apply our iterative k-means searchlight followed by subset scanning. Several anomalous trends in the data are identified, including cost overruns at specific hospitals, and increases in diagnoses such as suicides. These constitute novel findings in the literature, and are of potential use to regulatory agencies, policy makers and concerned citizens.

Our brain is wired to handle sensory inputs from multiple modalities such as vision, hearing and touch. Computational models have largely focused on a single sensory channel at a time, most notably vision and audition. Hence there is a gap in our understanding of the mechanisms that govern the integrated processing of information in these channels. One approach that has received attention in the literature is sensory binding achieved through synchronization. We demonstrate that synchronization can be realized through a network of oscillatory elements. We examine factors governing synchronization, specifically the tuning function that determines the sensitivity of the oscillators to phase differences. We develop a computational model that allows the exploration of phenomena studied in cognitive neuroscience, including deficits in synchronization that accompany disorders such as Attention Deficit Hyperactivity Disorder. The width of the tuning function affects the ability of subjects to combine auditory and visual streams of information, where broader tuning functions are associated with poorer performance. We demonstrate that quantitative measures of simulated system performance behave similarly to data gathered by cognitive neuroscience experimentalists. Thus, we produce results that are consistent with findings in cognitive neuroscience and psychology. Our model could serve as a basis to understand the subtle dynamics of processes involved in both normal and abnormal perception. A quantitative approach could help researchers model the effect of different interventions, including drugs such as methylphenidate that promote synchrony.

When analyzing data where the relationships between variables are not fully understood, it is typical to engage in visual exploration. However, this is slow and manually intensive, and interesting trends could potentially be missed. Ideally, we would like to focus on patterns and locations within the data that are interesting, such as significant clusters and outliers. We present a novel iterative k-means clustering algorithm to efficiently identify clusters in large datasets. This facilitates rapid visual exploration of new datasets. We illustrate our framework by performing a detailed analysis of open health care data released by the US Government and New York State. We apply our iterative k-means algorithm to identify clusters of trends in labor force participation to present a unique perspective by area of medical specialty over a 50-year period. Specialties such as nurse practitioners have seen a significant increase in the number of practitioners relative to internal medicine. Our code is open source and available on Github.

Since the world we live in consists of objects that can be experienced through multiple senses simultaneously, there are advantages offered by a multisensory learning paradigm. Though advances are being made in the fields of neuroscience, cognitive science and psychology regarding multisensory learning, there is a gap that exists between these advances and the ability of current computational models to explain them. There are relatively few computational models in the literature that directly address multisensory learning, while taking into account aspects surrounding temporal synchrony. In order to address this gap, we present a principled approach to understanding multisensory learning, based on the concept of sparse spatio-temporal encoding. We bring our model into closer alignment with recently published results in neuroscience and cognitive science. We demonstrate that multisensory learning produces greater and more efficient learning. Furthermore, we show that the use of non-congruent stimuli results in significantly worse performance during recall. When a sensory channel becomes disabled, we show that the network performance also degrades, but the degradation is less than that experienced during the presentation of non-congruent stimuli. Hence, our computational model produces experimental results that are consistent with observations in neuroscience, cognitive science and psychology. This research furthers our understanding of fundamental brain processes, which paves the way for multiple advances including the building of machines with more human-like capabilities.

Many government agencies worldwide are releasing public datasets about the services provided in healthcare, such as Medicare in the US. These datasets can be queried by multiple classes of users, including hospitals, patients, physicians and policy makers. However, to realize the true value of the information present in these datasets, appropriate analysis, including classification and clustering needs to be performed. Additional insight can be gained by combining the healthcare data with other data sources such as demographics and epidemiology. We provide an analytics framework based on open-source tools that facilitate the exploration of these datasets. We illustrate our framework by providing a detailed analysis of physician and hospital ratings data. Our technique should prove valuable to hospital administrators and policy makers.

The external world consists of objects that stimulate multiple sensory pathways simultaneously, such as auditory, visual and touch. Our brains receive and process these sensory streams to arrive at a coherent internal representation of the world. Though much attention has been paid to these streams individually, their integration is comparatively less well understood. In this paper we propose the principle of sparse spatio-temporal encoding as a foundation to build a framework for multi-sensory integration. We derive the dynamics that govern a network of oscillatory units that achieves phase synchronization, and is capable of binding related attributes of objects. We simulate objects that produce simultaneous visual and auditory input activations. We demonstrate that our system can bind features in both these sensory modalities. We examine the effect of varying a tuning function that governs the ability of the units to synchronize, and show that by broadening this function we reduce the ability of the network to disambiguate mixtures of objects. Thus, our model offers the potential to study brain disorders such as autism, which may arise from a disruption of synchronization.

Psychological studies have investigated limits in the capacity to simultaneously store multiple objects in working memory, which turns out to be approximately four. In this paper we examine the existence and origin of such a capacity limit in the sensory encoding stage, where synchronous activity can be considered to group related features of a common object. We develop a model of an object recognition network using oscillatory elements that can achieve phase synchronization. Using simulations based on this network, we show that distinct phases of oscillation can be used to label combinations of objects presented simultaneously. This allows the network to separate mixtures of objects, and identify the input elements that belong to each object. We demonstrate that there is a limit of four objects that can be separated from a mixture. Further studies are required to generalize this result by varying the size of the network and the number of objects used for training. We also show that by narrowing a tuning function that governs the dynamics of the system, we can achieve higher separation accuracy. However, this comes at the cost of utilizing a higher number of iterations over which the system settles and learns its synaptic weights. We lay down a framework for the quantitative modeling of factors affecting capacity limits, which has the potential to advance our understanding of sensory representation, attention, working memory and multitasking.

We present a computational model of the visual cortex using anatomically realistic projections incorporating two color opponent channels and a luminance channel. We model the visual cortex area V1 with a red-green opponent channel projecting to a simulated layer 4c beta, and a luminance channel projecting to layer 4c alpha. Both these layers project to a cortical layer 2/3. We specifically model the konio-cellular pathway, providing direct blue-yellow chromatic input to layer 2/3 within targeted blob regions. We utilize a self-organizing map algorithm to learn synaptic weights while the system is driven by artificial inputs initially, and followed by natural image stimuli. The same algorithm is used for learning in all the hierarchically structured maps. We produce maps of orientation and color selectivity, and apply statistical measures to examine the variations in these maps between the blob and interblob regions. We show that an orientation map exists throughout layer 2/3, whereas a complete color hue map is produced in the blob regions. Both orientation and color selectivity are present throughout layer 2/3. The mean orientation selectivity is lower in the blob regions as compared to the interblob regions. However, there is greater color selectivity in the blob regions compared to the interblob regions. We make a testable prediction that a statistically significant difference in color selectivity in neuronal populations will be observed in the primate visual cortex between blob and interblob regions.

We model neural dynamical behavior during object perception using the principle of sparse coding in multilayer oscillatory networks. The network model consists of units with amplitude and phase variables, and allows the propagation of higher-level information to lower levels via feedback connections. We show that this model can replicate findings in the neuroscience literature, where measurements have shown that neurons in lower level visual areas respond in a delayed fashion to missing contours of whole objects. We contrast the behavior of feedback connections with that of lateral connections by selectively disabling these in our model to examine their contributions to object perception. This paper successfully extends the previously reported capabilities of oscillatory networks by applying them to model perceptual tasks.

The visual cortex contains well-ordered maps for visual features such as orientation. Existing computational approaches have demonstrated that biologically realistic global organization can be obtained. However, specific local properties such as variability in orientation tuning have not been thoroughly investigated. The purpose of this paper is to provide a computationally grounded interpretation of the evolution of topographic properties of cortical maps. We use a modified Kohonen approach to self-organization, and show that the orientation tuning of cortical units varies such that orientation selectivity is weaker at pinwheel centers than linear zones. This finding impacts a debate in neuroscience, where researchers are trying to resolve whether or not orientation tuning is sharp everywhere. We also examined a null-hypothesis, which modified the behavior of cortical units so that they are free to choose their orientation preference, but keep their orientation tuning constant. Our approach offers a mechanistic interpretation for how local learning algorithms can shape global organization. Our results show that a spatially differentiated orientation tuning map is intrinsic to a spatial learning process. Furthermore, constraining the orientation tuning of cortical units to be constant results in unacceptable map formation.

The relationship between structure and function is of central importance in neuroscience. Computational modeling techniques can play a crucial role in exploring this relationship. Neuroscientists have revealed an interesting patterning in the connectivity of visual cortical areas, where the receptive field sizes for feed-forward, lateral and feedback connections are monotonically increasing and these roughly double. In this paper, we use a computational modeling approach to understand the behavior of the visual system, and show that this observed connectivity pattern can be explained via a maximization of functional metrics based on separation and segmentation accuracies. We use an optimization function based on sparse spatiotemporal encoding and faithfulness of representation to derive the dynamical behavior of a multi-layer network of oscillatory units. The network behavior can be quantified in terms of its ability to separate and segment mixtures of inputs. We vary the topological connectivity between different layers of the simulated visual network and study its effect on performance in terms of separation and segmentation accuracy. We demonstrate that the best performance occurs when the topology of the simulated system is similar to that in the primate visual cortex where the receptive field sizes of feedforward, lateral and feedback connections are monotonically increasing. This explanation of the functional significance of topological connectivity provides a new perspective for the understanding of cortical function.

In earlier work, we derived the dynamical behavior of a network of oscillatory units described by the amplitude and phase of oscillations. The dynamics were derived from an objective function that rewards both the faithfulness and the sparseness of representation. After unsupervised learning, the network is capable of separating mixtures of inputs, and also segmenting the inputs into components that most contribute to the classification of a given input object. In the current paper, we extend our analysis to multi-layer networks, and demonstrate that the dynamical equations derived earlier can be successfully applied to multi-layer networks. The topological connectivity between the different layers are derived from biological observations in primate visual cortex, and consist of receptive fields that are topographically mapped between layers. We explore the role of feedback connections, and show that increasing the diffusivity of feedback connections significantly improves segmentation performance, but does not affect separation performance.

In this paper, we present a novel network to separate mixtures of inputs that have been previously learned. A significant capability of the network is that it segments the components of each input object that most contribute to its classification. The network consists of amplitude-phase units that can synchronize their dynamics, so that separation is determined by the amplitude of units in an output layer, and segmentation by phase similarity between input and output layer units. Learning is unsupervised and based on a Hebbian update, and the architecture is very simple. Moreover, efficient segmentation can be achieved even when there is considerable superposition of the inputs. The network dynamics are derived from an objective function that rewards sparse coding in the generalized amplitude-phase variables. We argue that this objective function can provide a possible formal interpretation of the binding problem and that the implementation of the network architecture and dynamics is biologically plausible.

We present a novel network of oscillatory units, whose behavior is described by the amplitude and phase of oscillations. We derive the network dynamics from an objective function that rewards both the faithfulness and the sparseness of representation. The resulting network architecture is simple, and the dynamics are straightforward to interpret. This network functions in an unsupervised manner, and is able to form unique representations for a set of inputs. Once the set of inputs is learnt, the network can deconvolve mixtures of inputs. A significant capability of the network is that it segments its inputs into components that most contribute to the classification of a given input object. Network units exhibit synchronization through phase locking after an initial settling period. The behavior of deconvolution is determined by the amplitude of units in an output layer, while segmentation is simultaneously determined through phase similarity between the input and output layer units. Thus, the network exhibits the binding of specific input units and the output units that represent a classification of these input units by means of phase similarity.

The advent of electronic publication creates strong interest in converting existing printed documents into electronic formats. During this process, image reproduction problems can occur due to the formation of Moire patterns in the screened halftone areas. Therefore, the optimal quality of a scanned document is achieved if halftone regions are first identified and processed separately. We propose a complete algorithm to achieve this objective. A wavelet based halftone segmentation algorithm is first designed to locate possible halftone regions using a decision function. We then introduce a suboptimal FIR descreening filter to efficiently handle various screening frequencies and angles. Experimental results are offered to illustrate the performance of our algorithm.

The advent of electronic publishing creates strong interest in converting existing printed documents into electronic formats. During this process, image reproduction problems can occur due to the formation of moire patterns in the screened halftone areas. A wavelet packet based color halftone segmentation algorithm is first designed to locate possible halftone regions based on a decision function. A color halftone descreening technique based on the color sigma filter is proposed which does not assume any a priori knowledge about the halftoning process, making it applicable to any color halftone image. Combined with color halftone segmentation techniques, a complete document processing algorithm for color documents is proposed. Experimental results are offered to illustrate the performance of our algorithm.

In this paper, we address the requirements, techniques and applications of digital watermarking for high-quality images. We present several classes of digital watermarking techniques defined by their appearance and application domains, specify the requirements, and summarize the current research efforts in the Image Library Applications group of the IBM Thomas J. Watson Research Center on digital watermarking of high-quality images. We hope that by addressing these issues, we can both provide a foundation for the next generation of digital watermarking techniques, and a vision for their future applications.

Recently, researchers have started using texture for data visualization. The rationale behind this is to exploit the sensitivity of the human visual system to texture in order to overcome the limitations inherent in the display of multidimensional data. A fundamental issue that must be addressed is what textural features are important in texture perception, and how they are used. We designed an experiment to help identify the relevant higher order features of texture perceived by humans. We used twenty subjects, who were asked to rate 56 pictures from Brodatz's album on 12 nine-point Likert scales. We applied the techniques of hierarchical cluster analysis, non-parametric multidimensional scaling (MDS), classification and regression tree analysis (CART), discriminant analysis, and principal component analysis to data gathered from the subjects. Based on these techniques, we identified three orthogonal dimensions for texture to be repetitive vs. non-repetitive; high-contrast and non-directional vs. low-contrast and directional; granular, coarse and low-complexity vs. non-granular, fine and high-complexity.>

A fundamental issue in texture analysis is that of deciding what textural features are important in texture perception, and how they are used. Experiments on human preattentive vision have identified several low-level features (such as orientation of blobs and size of line segments), which are used in texture perception. However, the question of what higher level features of texture are used has not been adequately addressed. We designed an experiment to help identify the relevant higher order features of texture perceived by humans. We used 20 subjects, who were asked to perform an unsupervised classification of 30 pictures from Brodatz′s album on texture. Each subject was asked to group these pictures into as many classes as desired. Both hierarchical cluster analysis and nonparametric multidimensional scaling (MDS) were applied to the pooled similarity matrix generated from the subjects′ groupings. A surprising outcome is that the MDS solutions fit the data very well. The stress in the two-dimensional case is 0.10, and the stress in the three-dimensional case is 0.045. We rendered the original textures in these coordinate systems, and interpreted the (rotated) axes. It appears that the axes in the 2D case correspond to periodicity versus irregularity, and directionality versus nondirectionality. In the 3D case, the third dimension represents the structural complexity of the texture. Furthermore, the clusters identified by the hierarchical cluster analysis remain virtually intact in the MDS solution. The results of our experiment indicate that people use three high-level features for texture perception. Future studies are needed to determine the appropriateness of these high-level features for computational texture analysis and classification.

Confocal imaging is an emerging technique for the measurement of surface topography in inspection. The authors present a system designed for fast acquisition and processing of confocal images, which consists of an optical front end using tilted confocal scanning, and an image processing module. The function of the image processing module is to improve signal resolution, perform smoothing and detect surfaces in the incoming signal. The input signal is first deconvolved in order to improve the depth resolution, and then processed to identify significant peaks. These peaks represent the position of different surfaces in the object being inspected. These peak locations are smoothed using a cluster based smoothing scheme to combat noise. For semi-transparent materials, the authors system is capable of detecting up to two surfaces at a given location.>

An approach to the solution of signal-to-symbol transformation in the domain of flow fields, such as oriented texture fields and velocity vector fields, is discussed. The authors use the geometric theory of differential equations to derive a symbol set based on the visual appearance of phase portraits which are a geometric representation of the solution curves of a system of differential equations. They also provide the computational framework to start with a given flow field and derive its symbolic representation. Specifically, they segment the given texture, derive its symbolic representation, and perform a quantitative reconstruction of the salient features of the original texture based on the symbolic descriptors. Results of applying this technique to several real texture images are presented. This technique is useful in describing complex flow visualization pictures, defects in lumber processing, defects in semiconductor wafer inspection, and optical flow fields.>

An attempt is made to develop a solution for signal-to-symbol transformation in the domain of flowlike or oriented texture. The geometric theory of differential equations is used to derive a symbol set based on the visual appearance of phase portraits. This theory provides a technique for describing textures both qualitatively and quantitatively. An attractive feature of this symbol set is that it is domain independent and makes no assumptions about the kind of texture that may be present. The computational framework for starting with a given oriented texture is provided, and its symbolic representation is derived. This is based on computing the orientation field for the texture and then using a nonlinear least-squares technique over successive windows to determine the changing spatial behavior of the texture. Results of the application of this technique to real texture images are presented.>

A novel technique to analyze stereo images generated from a scanning electron microscope (SEM) for inspection of wafers for process control is presented. The two main features of this technique are that it uses a binary linear programming approach to set up and solve the correspondence problem, and that it uses constraints based on the physics of SEM image formation. Binary linear programming is a powerful tool with which to tackle constrained optimization problems, especially in the cases that involve matching between one data set and another. The authors also analyze the process of SEM image formation, and present constraints that are useful in solving the stereo correspondence problem. This technique has been solved on many images. Results for a few wafers are included.>

A novel algorithm for computing the orientation field for a flowlike texture is presented. The basic idea behind the algorithm is to use an oriented filter, namely the gradient of the Gaussian, and perform manipulations on the resulting gradient vector field. The most important aspect of the algorithm is that it is provably optimal in estimating the local orientations of an oriented texture. An added strength of the algorithm is that it is simpler and has a better signal-to-noise ratio than previous approaches, because it utilizes fewer derivative operations. Also proposed is a measure of coherence which is based on the use of the angle and coherence images as intrinsic images. An analysis of oriented textures requires the computation of these intrinsic images as a first step. In this sense, the computation of the orientation field, resulting in the intrinsic images, is indispensable in the analysis of oriented textures. Results from a number of experiments indicate the usefulness of the angle and coherence intrinsic images.>