This article investigates the application of deep learning (DL) to the fusion of omnidirectional (O-D) infrared (IR) sensors and O-D visual sensors to improve the intelligent perception of autonomous robotic systems. Recent techniques primarily focus on O-D and conventional visual sensors for applications in localization, mapping, and tracking. The robotic vision systems have not sufficiently utilized the combination of O-D IR and O-D visual sensors, coupled with DL, for the extraction of vegetation material. We will be showing the contradiction between current approaches and our deep vegetation learning sensor fusion. This article introduces two architectures: 1) the application of two autoencoders feeding into a four-layer convolutional neural network (CNN) and 2) two deep CNN feature extractors feeding a deep CNN fusion network (DeepFuseNet) for the fusion of O-D IR and O-D visual sensors to better address the number of false detects inherent in indices-based spectral decomposition. We compare our DL results to our previous work with normalized difference vegetation index (NDVI) and IR region-based spectral fusion, and to traditional machine learning approaches. This work proves that the fusion of the O-D IR and O-D visual streams utilizing our DeepFuseNet DL approach outperforms both the previous NVDI fused with far-IR region segmentation and traditional machine learning approaches. Experimental results of our method validate a 92% reduction in false detects compared to traditional indices-based detection. This article contributes a novel method for the fusion of O-D visual and O-D IR sensors using two CNN feature extractors feeding into a deep CNN (DeepFuseNet).

Cloud Colonography is proposed in this paper, using different types of cloud computing environments. Databases from the Computed Tomographic Colonography (CTC) screening tests among several hospitals are explored. These networked databases are going to be available in the near future via cloud computing technologies. Associated Multiple Databases (AMD) was developed in this study to handle multiple CTC databases. When AMD is used for assembling databases, it can achieve very high classification accuracy. The proposed AMD has the potential to play a role of the core classifier in the cloud computing framework. AMD for multiple institutions databases yields high detection performance of polyps using Kernel principal component analysis (KPCA). Two cases in the proposed cloud platform are private and public. We adapted a university cluster as a private platform, and Amazon Elastic Compute Cloud (EC2) as a public. The computation time, memory usage, and running costs were compared using three representative databases between private and public cloud environments. The proposed parallel processing modules improved the computation time, especially in the public cloud environments. The successful development of a cloud computing environment that handles large amounts of data will make Cloud Colonography feasible for a new health care service.

When detecting infrared maritime targets on sunny days, strong sun glitters can lower the detection accuracy tremendously. To this problem, we proposed a robust antijitter spatiotemporal saliency generation with parallel binarization (ASSGPB) method. Its main contribution is to facilitate to improve the infrared maritime target detection accuracy in this situation. The ASSGPB algorithm exploits the target's spatial saliency and temporal consistency to separate real targets from clutter areas. The ASSGPB first corrects image intensity distribution with a central inhibition difference of Gaussian filter. Then, a self-defined spatiotemporal saliency map (STSM) generator is used to generate an STSM in five consecutive frames while compensating interframe jitters by a joint block matching. Finally, a parallel binarization method is adopted to segment real targets in STSM while keeping full target areas. To evaluate the performance of ASSGPB, we captured eight different image sequences (20420 frames in total) that were significantly contaminated by strong sun glitters. The ASSGPB realized 100% detection rate and 0.45% false alarm rate in these data sets, greatly outperforming four state-of-the-art algorithms. Thus, a great applicability of ASSGPB has been verified through our experiments.

Collaborative robotic configurations for monitoring and tracking human targets have attracted interest in the fourth industrial revolution. The fusion of different types of sensors embedded in collaborative robotic systems achieves high-quality information and contributes to significantly improve robotic perception. However, current methods have not deeply explored the capabilities of thermal multisensory configurations in human-oriented tasks. In this paper, we propose thermal multisensor fusion (TMF) for collaborative robots to overcome the limitations of stand-alone robots. Thermal vision helps to utilize the heat signature of the human body for human-oriented tracking. An omnidirectional (O-D) infrared (IR) sensor provides a wide field of view (FOV) to detect human targets, and Stereo IR helps determine the distance of the human target in the oriented direction. The fusion of O-D IR and Stereo IR also creates a multisensor stereo for an additional determination of the distance to the target. The fusion of thermal and O-D sensors brings their limited prediction accuracy with their advantages. The maximum a posteriori method is used to predict the distance of the target with high accuracy by using the distance results of TMF stereo from multiple platforms according to the reliability of the sensors rather than its usage of visible-band-based tracking methods. The proposed method tracks the distance calculation of each sensor instead of target trajectory tracking as in visible-band methods. We proved that TMF increases the perception of robots by offering a wide FOV and provides precise target localization for collaborative robots.

This paper proposes the multicolumn RBF network (MCRN) as a method to improve the accuracy and speed of a traditional radial basis function network (RBFN). The RBFN, as a fully connected artificial neural network (ANN), suffers from costly kernel inner-product calculations due to the use of many instances as the centers of hidden units. This issue is not critical for small datasets, as adding more hidden units will not burden the computation time. However, for larger datasets, the RBFN requires many hidden units with several kernel computations to generalize the problem. The MCRN mechanism is constructed based on dividing a dataset into smaller subsets using the k-d tree algorithm. resultant subsets are considered as separate training datasets to train individual RBFNs. Those small RBFNs are stacked in parallel and bulged into the MCRN structure during testing. The MCRN is considered as a well-developed and easy-to-use parallel structure, because each individual ANN has been trained on its own subsets and is completely separate from the other ANNs. This parallelized structure reduces the testing time compared with that of a single but larger RBFN, which cannot be easily parallelized due to its fully connected structure. Small informative subsets provide the MCRN with a regional experience to specify the problem instead of generalizing it. The MCRN has been tested on many benchmark datasets and has shown better accuracy and great improvements in training and testing times compared with a single RBFN. The MCRN also shows good results compared with those of some machine learning techniques, such as the support vector machine and k-nearest neighbors.

Accurate sorting of beam projections is important in 4D cone beam computed tomography (4D CBCT) to improve the quality of the reconstructed 4D CBCT image by removing motion-induced artifacts. We propose image registration-based projection binning (IRPB), a novel marker-less binning method for 4D CBCT projections, which combines intensity-based feature point detection and trajectory tracking using random sample consensus. IRPB extracts breathing motion and phases by analyzing tissue feature point trajectories. We conducted experiments with two phantom and six patient datasets, including both regular and irregular respirations. In experiments, we compared the performance of the proposed IRPB, Amsterdam Shroud method (AS), Fourier transform-based method (FT), and local intensity feature tracking method (LIFT). The results showed that the average absolute phase shift of IRPB was 3.74 projections and 0.48 projections less than that of FT and LIFT, respectively. AS lost the most breathing cycles in the respiration extraction for the five patient datasets, so we could not compare the average absolute phase shift between IRPB and AS. Based on the peak signal-to-noise ratio (PSNR) of the reconstructed 4D CBCT images, IRPB had 5.08, 1.05, and 2.90 dB larger PSNR than AS, FT, and LIFT, respectively. The average Structure SIMilarity Index (SSIM) of the 4D CBCT image reconstructed by IRPB, AS, and LIFT were 0.87, 0.74, 0.84, and 0.70, respectively. These results demonstrated that IRPB has superior performance to the other standard methods.

A method for integrating, processing, and analyzing sensing data from vehicle-mounted sensors for intelligent forecasting and decision-making is introduced. This dead reckoning with dynamic errors (DRWDEs) is for a large-scale integration of distributed resources and sensing data intervehicle collision avoidance system. This sensor fusion algorithm is introduced to predict the future trajectory of a vehicle. Current systems that predict a vehicle's future trajectory, necessary in a network of collision avoidance systems, tend to have a lot of errors when the vehicles are moving in a nonstraight path. Our system has been designed with the objective of improving the estimations during curves. To evaluate this system, our research uses a Garmin 16HVS GPS sensor, an AutoEnginuity OBDII ScanTool, and a Crossbow three-axis accelerometer. Using Kalman filters (KFs), a dynamic noise covariance matrix merged together with an interacting multiple models (IMMs) system, our DRWDE produces the future position estimation of where the vehicle will be 3 s later in time. The ability to handle the change in noise, depending on unavailable sensor measurements, permits a flexibility to use any type of sensor and still have the system run at the fastest frequency available. Compared with a more common KF implementation that runs at the rate of its slowest sensor (1 Hz in our setup), our experimental results showed that our DRWDE (running at 10 Hz) yielded more accurate predictions (25%-50% improvement) during abrupt changes in the heading of the vehicle.

To enhance the quality of economically efficient healthcare, we propose a preventive planning service for next-generation screening based on a longitudinal prediction. This newly proposed framework may bring important advancements in prevention by identifying the early stages of cancer, which will help in further diagnoses and initial treatment planning. The preventive service may also solve the obstacles of cost and availability of scanners in screening. For nonstationary medical data, anomaly detection is the key problem in the prediction of cancer staging. To address anomaly detection in a huge stream of databases, we applied a composite kernel to the prediction of cancer staging for the first time. The proposed longitudinal analysis of composite kernels (LACK) is designed for the prediction of anomaly status and cancer stage for further diagnosis and the future likelihood of cancer stage progression. The prediction error of LACK is relatively small even if the prediction is made far ahead of time. The computation time for nonstationary learning is reduced by 33% compared with stationary learning.

Computer-Aided Detection (CAD) of polyps in Computed Tomographic (CT) colonography is currently very limited since a single database at each hospital/institution doesn't provide sufficient data for training the CAD system's classification algorithm. To address this limitation, we propose to use multiple databases, (e.g., big data studies) to create multiple institution-wide databases using distributed computing technologies, which we call smart colonography. Smart colonography may be built by a larger colonography database networked through the participation of multiple institutions via distributed computing. The motivation herein is to create a distributed database that increases the detection accuracy of CAD diagnosis by covering many true-positive cases. Colonography data analysis is mutually accessible to increase the availability of resources so that the knowledge of radiologists is enhanced. In this article, we propose a scalable and efficient algorithm called Group Kernel Feature Analysis (GKFA), which can be applied to multiple cancer databases so that the overall performance of CAD is improved. The key idea behind the proposed GKFA method is to allow the feature space to be updated as the training proceeds with more data being fed from other institutions into the algorithm. Experimental results show that GKFA achieves very good classification accuracy.

Kernel association (KA) in statistical pattern recognition used for classification and prediction have recently emerged in a machine learning and signal processing context. This survey outlines the latest trends and innovations of a kernel framework for big data analysis. KA topics include offline learning, distributed database, online learning, and its prediction. The structural presentation and the comprehensive list of references are geared to provide a useful overview of this evolving field for both specialists and relevant scholars.

Virtual reality and augmented reality environments using helmet-mounted displays create a sense of immersion by closely coupling user head motion to display content. Delays in the presentation of visual information can destroy the sense of presence in the simulation environment when it causes a lag in the display response to user head motion. The effect of display lag can be minimized by predicting head orientation, allowing the system to have sufficient time to counteract the delay. In this paper, anew head orientation prediction technique is proposed that uses a multiple delta quaternion (DQ) extended Kalman filter to track angular head velocity and angular head acceleration. This method is independent of the device used for orientation measurement, relying on quaternion orientation as the only measurement data. A new orientation prediction algorithm is proposed that estimates future head orientation as a function of the current orientation measurement and a predicted change in orientation, using the velocity and acceleration estimates. Extensive experimentation shows that the new method improves head orientation prediction when compared to single filter DQ prediction.

Tracking human motion with multiple body sensors has the potential to promote a large number of applications such as detecting patient motion, and monitoring for home-based applications. With multiple sensors, the tracking system architecture and data processing cannot perform the expected outcomes because of the limitations of data association. For the collaborative and intelligent applications of motion tracking (Polhemus Liberty AC magnetic tracker), we propose a human motion tracking system with multichannel interacting multiple model estimator (MC-IMME). To figure out interactive relationships among distributed sensors, we used a Gaussian mixture model (GMM) for clustering. With a collaborative grouping method based on GMM and expectation-maximization algorithm for distributed sensors, we can estimate the interactive relationship with multiple body sensors and achieve the efficient target estimation to employ a tracking relationship within a cluster. Using multiple models with filter divergence, the proposed MC-IMME can achieve the efficient estimation of the measurement and the velocity from measured datasets of human sensory data. We have newly developed MC-IMME to improve overall performance with a Markov switch probability and a proper grouping method. The experiment results shows that the prediction overshoot error can be improved on average by 19.31% by employing a tracking relationship.

Information processing of radiotherapy systems has become an important research area for sophisticated radiation treatment methodology. Geometrically precise delivery of radiotherapy in the thorax and upper abdomen is compromised by respiratory motion during treatment. Accurate prediction of the respiratory motion would be beneficial for improving tumor targeting. However, a wide variety of breathing patterns can make it difficult to predict the breathing motion with explicit models. We proposed a respiratory motion predictor, that is, customized prediction with multiple patient interactions using neural network (CNN). For the preprocedure of prediction for individual patient, we construct the clustering based on breathing patterns of multiple patients using the feature selection metrics that are composed of a variety of breathing features. In the intraprocedure, the proposed CNN used neural networks (NN) for a part of the prediction and the extended Kalman filter (EKF) for a part of the correction. The prediction accuracy of the proposed method was investigated with a variety of prediction time horizons using normalized root mean squared error (NRMSE) values in comparison with the alternate recurrent neural network (RNN). We have also evaluated the prediction accuracy using the marginal value that can be used as the reference value to judge how many signals lie outside the confidence level. The experimental results showed that the proposed CNN can outperform RNN with respect to the prediction accuracy with an improvement of 50%.

Principal composite kernel feature analysis (PC-KFA) is presented to show kernel adaptations for nonlinear features of medical image data sets (MIDS) in computer-aided diagnosis (CAD). The proposed algorithm PC-KFA has extended the existing studies on kernel feature analysis (KFA), which extracts salient features from a sample of unclassified patterns by use of a kernel method. The principal composite process for PC-KFA herein has been applied to kernel principal component analysis [34] and to our previously developed accelerated kernel feature analysis [20]. Unlike other kernel-based feature selection algorithms, PC-KFA iteratively constructs a linear subspace of a high-dimensional feature space by maximizing a variance condition for the nonlinearly transformed samples, which we call data-dependent kernel approach. The resulting kernel subspace can be first chosen by principal component analysis, and then be processed for composite kernel subspace through the efficient combination representations used for further reconstruction and classification. Numerical experiments based on several MID feature spaces of cancer CAD data have shown that PC-KFA generates efficient and an effective feature representation, and has yielded a better classification performance for the proposed composite kernel subspace using a simple pattern classifier.

Complicated breathing behaviors including uncertain and irregular patterns can affect the accuracy of predicting respiratory motion for precise radiation dose delivery [3-6, 25, 36]. So far investigations on irregular breathing patterns have been limited to respiratory monitoring of only extreme inspiration and expiration [37]. Using breathing traces acquired on a Cyberknife treatment facility, we retrospectively categorized breathing data into several classes based on the extracted feature metrics derived from breathing data of multiple patients. The novelty of this paper is that the classifier using neural networks can provide clinical merit for the statistical quantitative modeling of irregular breathing motion based on a regular ratio representing how many regular/irregular patterns exist within an observation period. We propose a new approach to detect irregular breathing patterns using neural networks, where the reconstruction error can be used to build the distribution model for each breathing class. The proposed irregular breathing classification used a regular ratio to decide whether or not the current breathing patterns were regular. The sensitivity, specificity, and receiver operating characteristic (ROC) curve of the proposed irregular breathing pattern detector was analyzed. The experimental results of 448 patients breathing patterns validated the proposed irregular breathing classifier.

AC electromagnetic trackers are well suited for head tracking but are adversely affected by conductive and ferromagnetic materials. Tracking performance can be improved by mapping the tracking volume to produce coefficients that correct position and orientation (PnO) measurements caused by stationary distorting materials. The mapping process is expensive and time consuming, requiring complicated high-precision equipment to provide registration of the measurements to the source reference frame. In this study, we develop a new approach to mapping that provides registration of mapping measurements without precision equipment. Our method, i.e., the interpolation volume calibration system, uses two simple fixtures, each with multiple sensors in a rigid geometry, to determine sensor PnO in a distorted environment without mechanical measurements or other tracking technologies. We test our method in a distorted tracking environment, constructing a lookup table of the magnetic field that is used as the basis for distortion compensation. The new method compares favorably with the traditional approach providing a significant reduction in cost and effort.

Display lag in simulation environments with helmet-mounted displays causes a loss of immersion that degrades the value of virtual/augmented reality training simulators. Simulators use predictive tracking to compensate for display lag, preparing display updates based on the anticipated head motion. This paper proposes a new method for predicting head orientation using a delta quaternion (DQ)-based extended Kalman filter (EKF) and compares the performance to a quaternion EKF. The proposed framework operates on the change in quaternion between consecutive data frames (the DQ), which avoids the heavy computational burden of the quaternion motion equation. Head velocity is estimated from the DQ by an EKF and then used to predict future head orientation. We have tested the new framework with captured head motion data and compared it with the computationally expensive quaternion filter. Experimental results indicate that the proposed DQ method provides the accuracy of the quaternion method without the heavy computational burden.

A fast algorithm, Accelerated Kernel Feature Analysis (AKFA), that discovers salient features evidenced in a sample of n unclassified patterns, is presented. Like earlier kernel-based feature selection algorithms, AKFA implicitly embeds each pattern into a Hilbert space, H, induced by a Mercer kernel. An \ell-dimensional linear subspace of H is iteratively constructed by maximizing a variance condition for the nonlinearly transformed sample. This linear subspace can then be used to define more efficient data representations and pattern classifiers. AKFA requires O(\elln2) operations, as compared to 0(n^3) for Sch¨olkof, Smola, and M¨uller’s Kernel Principal Component Analysis (KPCA), and O(\ell^2 n^2) for Smola, Mangasarian, and Sch¨olkopf’s Sparse Kernel Feature Analysis (SKFA). Numerical experiments show that AKFA can generate more concise feature representations than both KPCA and SKFA, and demonstrate that AKFA obtains similar classification performance as KPCA for a face recognition problem.

This paper presents a human-computer interaction (HCI) framework for building vision models of three-dimensional (3-D) objects from their two-dimensional (2-D) images. Our framework is based on two guiding principles of HCI: 1) provide the human with as much visual assistance as possible to help the human make a correct input; and 2) verify each input provided by the human for its consistency with the inputs previously provided. For example, when stereo correspondence information is elicited from a human, his/her job is facilitated by superimposing epipolar lines on the images. Although that reduces the possibility of error in the human marked correspondences, such errors are not entirely eliminated because there can be multiple candidate points close together for complex objects. For another example, when pose-to-pose correspondence is sought from a human, his/her job is made easier by allowing the human to rotate the partial model constructed in the previous pose in relation to the partial model for the current pose. While this facility reduces the incidence of human-supplied pose-to-pose correspondence errors, such errors cannot be eliminated entirely because of confusion created when multiple candidate features exist close together. Each input provided by the human is therefore checked against the previous inputs by invoking situation-specific constraints. Different types of constraints (and different human-computer interaction protocols) are needed for the extraction of polygonal features and for the extraction of curved features. We will show results on both polygonal objects and object containing curved features.

Viewpoint calibration is a method to manipulate hand-eye for generating calibration parameters for active viewpoint control and object grasping. This paper presents a new approach to hand-eye robotic calibration for vision-based object modeling and grasping. Our method provides a 1.0 pixel level of image registration accuracy when a standard Puma robot generates an arbitrary viewpoint. In order to attain this accuracy, our new formalism of hand-eye calibration deals with a lens distortion model of a vision sensor and utilizes a new parameter estimation algorithm using an extended Kalman filter. We demonstrate the power of this new method called "SmartView" for: 1) generating 3D object models using an interactive 3D modeling editor; 2) recognizing 3D objects using stereo vision systems; and 3) grasping 3D objects using a manipulator. Experimental results using a Puma robot are shown.