Camera-based passive dietary intake monitoring is able to continuously capture the eating episodes of a subject, recording rich visual information, such as the type and volume of food being consumed, as well as the eating behaviors of the subject. However, there currently is no method that is able to incorporate these visual clues and provide a comprehensive context of dietary intake from passive recording (e.g., is the subject sharing food with others, what food the subject is eating, and how much food is left in the bowl). On the other hand, privacy is a major concern while egocentric wearable cameras are used for capturing. In this article, we propose a privacy-preserved secure solution (i.e., egocentric image captioning) for dietary assessment with passive monitoring, which unifies food recognition, volume estimation, and scene understanding. By converting images into rich text descriptions, nutritionists can assess individual dietary intake based on the captions instead of the original images, reducing the risk of privacy leakage from images. To this end, an egocentric dietary image captioning dataset has been built, which consists of in-the-wild images captured by head-worn and chest-worn cameras in field studies in Ghana. A novel transformer-based architecture is designed to caption egocentric dietary images. Comprehensive experiments have been conducted to evaluate the effectiveness and to justify the design of the proposed architecture for egocentric dietary image captioning. To the best of our knowledge, this is the first work that applies image captioning for dietary intake assessment in real-life settings.

Recent advances in low-power electronics, wireless communication, and bioengineering have produced many forms of implantable and migrating medical devices (a “migrating device” travels within the body along a specific route). Among them, smart implants and ingestible electronic pills that perform diagnostic functions inside the human body play increasing roles in modern medicine. Continuous access to physiological and biological information in real time disrupts the traditional symptom-based diagnostic approach, and ushers in a new era of technology- and evidence-based medicine. Despite the high potential impacts of diagnostic implants and migrating devices, numerous problems in both technical and biological domains must be solved. This talk focuses on several key problems common to these classes of devices: How to make them small while provide them with sufficient power and wireless communication functions when they are deeply located within the body? There are many compelling reasons to miniaturize these devices. First, smaller devices in centimeter or millimeter scales are easier to deliver in vivo using non-invasive (e.g., swallowing) or minimally invasive means, which in turn reduce medical care cost and complication rates. Second, smaller devices can be placed within less accessible compartments inside organs or tubular structures (e.g., a digestive tract) without interfering with the body’s normal biologic functions. Third, smaller devices improve biocompatibility and immune tolerance due to a smaller biological footprint. Despite these advantages, it remains technical challenge to construct such devices. For over two decades, our laboratory has been investigating the power and communication problems in miniature implants and migrating devices. While the power supply for a migrating device can be a battery if the lifetime of the device is short, a depleted battery within a permanent or long-term implant must be replaced surgically which is painful and expensive. To solve the power supply problem, we have studied three alternatives to the battery power. Firstly, the body’s own energy source is harvested using a biofuel cell taking advantage of the electrochemical activity of biological cells [1]. Secondly, electrical current is delivered to an implant through biological tissues using the volume conduction property of the human body [1]. Thirdly, wireless power transfer is used in which a power-carrying radio frequency wave (between megahertz and gigahertz) is transmitted from the outside to the inside of the human body [2]. To minimize device size and implement the data communication function, we have investigated a dual-functional design in which the wireless power delivery link and the data communication link are shared. Our design supports both downlink and uplink communication. While the downlink is relatively easy to implement using a power delivery signal with modulated amplitude, the uplink from the inside device to an outside unit is difficult, especially when the device is deeply located within the body. We must overcome the problems of limited power source, small transceiver antenna and long communication distance through layers of tissues. Our solution is to use a special tapped coil and a simple passive circuit to produce a strong pulsed magnetic field (PMF) for signal transmission within the body [2]. The strong PMF enables a data uplink from the implant to outside world in a high instantaneous transmission wattage through the shared power delivery channel, whereas a low average power consumption is maintained. Our experimental result showed that, when compared to the existing load shift keying (LSK) method, our method provides extraordinary improvements, with over 10 times increase in emitted signal strength and over 15 times increase in received signal strength.

The rapid development of autonomous driving in recent years presents many challenges for scene understanding. As an essential step towards scene understanding, semantic segmentation has received increased attention in the past few years. Although deep learning based approaches have achieved great success in improving the segmentation accuracy, most of them suffer from an inefficiency problem and can hardly be applied to real-time applications. In this paper, we analyze the computational cost of Convolutional Neural Network (CNN) and find that the inefficiency of CNNs is mainly caused by their wide structure rather than deep structure. In addition, the success of pruning based model compression methods proves that there are many redundant channels in CNNs. Thus, we design a narrow while deep backbone network to improve the efficiency of semantic segmentation. By casting our network to fully convolutional network (FCN32) segmentation architecture, the basic structure of most segmentation methods, we achieve 61.5% mIoU on Cityscapes validation dataset with only 4.2G floating-point operations (FLOPs) on 1024×2048 inputs, which already outperforms one of the earliest real-time deep learning based segmentation methods: ENet (58.3% mIoU, 3.8G FLOPs on 640×360 inputs). By further refining the output resolution of our network to the 1/8 of the input resolution with a simple encoder-decoder structure, we achieve 65.3% mIoU on Cityscapes test set with 14.0G FLOPs and 39.9 frames per second (FPS) on Titan X card. We have made our model publicly available at https://github.com/zgyang-hnu/NDNet.

Semantic segmentation is a key step in scene understanding for autonomous driving. Although deep learning has significantly improved the segmentation accuracy, current highquality models such as PSPNet and DeepLabV3 are inefficient given their complex architectures and reliance on multi-scale inputs. Thus, it is difficult to apply them to real-time or practical applications. On the other hand, existing real-time methods cannot yet produce satisfactory results on small objects such as traffic lights, which are imperative to safe autonomous driving. In this paper, we improve the performance of real-time semantic segmentation from two perspectives, methodology and data. Specifically, we propose a real-time segmentation model coined Narrow Deep Network (NDNet) and build a synthetic dataset by inserting additional small objects into the training images. The proposed method achieves 65.7% mean intersection over union (mIoU) on the Cityscapes test set with only 8.4G floatingpoint operations (FLOPs) on 1024×2048 inputs. Furthermore, by re-training the existing PSPNet and DeepLabV3 models on our synthetic dataset, we obtained an average 2% mIoU improvement on small objects.

Summary To reduce heavy noise from degraded video in low or predictable latency and preserve privacy, a powerful and efficient video denoising algorithm is proposed based on fog computing for Visual Internet of Things. The conventional method is to remove noise in the cloud; however, this may overload computation and communication and raise security and privacy issues. The proposed denoising algorithm is distributed to heterogeneous devices at network edges to preserve privacy and avoid security risks as noise can be reduced in the fog rather than the cloud. To address the problems of latency, communication rate, and extremely heavy noise, structure registration, inter‐frame and inner‐frame filters, and distribution compensation are applied in the proposed algorithm. A scheme for encrypting the denoised data at network edges is provided so that security and privacy issues may be avoided during transmission and storage. Compared with other denoising approaches under extremely heavy noise conditions, the experimental results demonstrate that the proposed approach achieves superior denoising performance in terms of peak signal‐noise ratio and visual quality at low computational cost, high bandwidth efficiency, and low‐latency response in a fog computing manner.

Action recognition in still images is a challenging task in computer vision. Recent successes in deep feature-learning advance this research, employing robust and rich-semantic feature representation. However, the issue that recognition fails when two action images share similar contexts is long-standing. In this paper, we employ metric learning method to address within-class and between-class confusions in action recognition. We propose a novel loss function, named composite-triplet loss. Supervised by this loss function, our method directly learns a similarity function from data. Employing a customized human action and context detection network, we obtain highly discriminative action image embeddings, which can be used in action image recognition and other tasks. Our approach is evaluated on the still-image action recognition task and the image caption generation task. On the PASCAL VOC dataset, our approach outperforms state-of-the-art methods, achieving 90.6% mean AP.

Visual context is fundamental to understand human actions in videos. However, to efficiently employ temporal context information presents an enormous challenge to this area. Two main problems are long-standing: (1) video frames are redundant while discriminative information is sparse; (2) large amount of interference information is mixed in frame sequences. These factors results in redundant computation and recognition failures. In this paper, we propose a learnable temporal attention mechanism to automatically select important time points from action sequences. We design an unsupervised Recurrent Temporal Sparse Autoencoder (RTSAE) network, which learns to extract sparse key-frames to sharpen discriminative yet to retain descriptive capability, as well to shield interfere information. By applying this technique to a recent proposed action recognition model Adaptive Recurrent-convolutional Hybrid network (ARCH), we significantly improve its performance in both speed and accuracy. Experiments demonstrate that, with the help of the RTSAE, ARCH outperforms most state-of-the-art methods on UCF101 and HMDB51 datasets.

A novel mask with a filled circle and outer ring is proposed to detect corners from images, based on the adaptive threshold of a local region. First, the inner filled circle is used in a response function to filter four non‐corner regions: image noise, object edges, corner neighbourhoods, and flat regions. Second, corner candidates are detected using a complex response function, by considering the margin of inner circle and the outer ring together. Finally, related algorithms are developed to determine and remove the false corners lying on thin‐band, noisy, and salient pixels. The authors’ approach has been tested on artificial, noisy, fuzzy, and real images, and its performance is evaluated, analysed, and compared with the existing grey‐level‐based corner detection methods of Harris, SUSAN, FAST, and Lan and Zhang. The presented corner detector has better detection accuracy, less sensitivity to noisy and fuzzy images, high computational efficiency, and good repeatability.

Recent advances in mobile devices have made profound changes in people's daily lives. In particular, the impact of easy access of information by the smartphone has been tremendous. However, the impact of mobile devices on healthcare has been limited. Diagnosis and treatment of diseases are still initiated by occurrences of symptoms, and technologies and devices that emphasize on disease prevention and early detection outside hospitals are under-developed. Besides healthcare, mobile devices have not yet been designed to fully benefit people with special needs, such as the elderly and those suffering from certain disabilities, such blindness. In this paper, an overview of our research on a new wearable computer called eButton is presented. The concepts of its design and electronic implementation are described. Several applications of the eButton are described, including evaluating diet and physical activity, studying sedentary behavior, assisting the blind and visually impaired people, and monitoring older adults suffering from dementia.

We propose a cross-trees structure to perform the non-local cost aggregation for dense stereo matching. The cross-trees structure consists of a horizontal-tree and a vertical-tree. Compared to other spanning trees, the significant superiority of the cross-trees is that the trees' constructions are efficient and independent on any local or global property. Moreover, the trees are exactly unique. By traversing the two crossed trees successively, a fast non-local cost aggregation algorithm is performed to filter the matching cost volume and then the disparity maps are established with the Winner-Take-All (WTA) strategy. Additionally, two different priors: edge prior and super pixel prior, are proposed to tackle the false smoothing at the depth boundaries. Hence, our method contains two different algorithms in terms of the cross-trees prior in this paper. Performance evaluation on the 27 Middlebury data sets shows that both our algorithms outperform the other two tree-based methods, namely minimum spanning tree (MST) and segment-tree (ST). By performing the non-local cost aggregation on different trees, MST, ST and our method all have competitive rankings on the Middlebury website compared to the local cost aggregation methods.

We derive an adaptive data-driven kernel in this paper to simultaneously address the kernel scale/orientation selection problem as well as the constant kernel shape in deformable object tracking applications. Level set technique is novelly introduced into the mean shift sample space to implement kernel evolution and update. Since the active contour model is designed to drive the kernel constantly to the direction that maximizes target likelihood, the kernel can adapt to target shape variation simultaneously with the mean shift iterations. Thus, it can give a better estimation bias to produce accurate shift of the mean and successfully avoid performance loss stemmed from pollution of the non-object regions hiding inside the kernel. Experimental results on a number of challenging sequences validate the effectiveness of the technique.

The complexity of human-machine interaction (HMI) is rapidly growing in modern industrial, medical, and military systems. Human operators are often challenged by control of high-order systems or unstable systems near the limits of controllability. However, there is no quantitative indication of human performance and cognitive workload in these difficult HMI tasks. Here, we characterize HMI as information flows measured in bits per second (b/s). We propose a control-theoretic framework to estimate the rate of information transmission in manual control. We demonstrate our method in an example of stabilizing an inverted pendulum, where we derive that, for a normal human operator, the information-transmission rate of manual control with one degree of freedom ranges between 3 and 4 b/s. This quantitative indication reveals the potential and limitation of human manual control and is instructive to the design of HMI interfaces that may maximally utilize human control commands.

In this study, joint source–channel decoding for non-binary source samples is conducted. The non-binary source samples can be modelled as the output of a multi-state Markov chain (MC). As the source samples are directly transmitted after channel coding without source compression, the transmitted signals can be highly correlated. At the receiver, the multi-state MC module can be designed to exploit the statistical correlation of source samples to improve the error correcting performance. However, as the number of states is increased, the multi-state MC module requires high computational complexity. To alleviate this problem, a simplified MC module is proposed. In the simplified MC module, the multi-state MC is replaced with multiple number of two-state MCs each of which exploits bit-level correlation of samples. Simulation results demonstrate that the simplified MC module can lead to competitive reduction in the required signal-to-noise ratio in comparison with the multi-state MC module with reduced computational complexity.

Postural synergies of the hand have been widely proposed in the literature, but only a few attempts were made to visualize temporal postural synergies, i.e., profiles of postural synergies varying over time. This paper aims to derive temporal postural synergies from kinematic synergies extracted from joint angular velocity profiles of rapid grasping movements. The rapid movements constrain the kinematic synergies to combine instantaneously, and thus, the movements can be approximated by a weighted summation of synchronous synergies. After being extracted by using singular value decomposition, the synchronous kinematic synergies were translated into temporal postural synergies, which revealed strategies of enslaving, metacarpal flexion for larger movements, and hierarchical recruitment of joints, adapted by subjects while grasping.

Recently, a highly efficient method to transmit power wirelessly using mid-range resonant coupling was reported. Based on this method, we present a multilayer thin film design of power transfer cells for medical applications. Consisting of a tape coil in the exterior layer and strips in the interior layer separated by an insulation layer, these cells have an equivalent structure of multiple inductors and capacitors, forming several resonant frequencies. In order to verify these frequencies, a mesh current analysis is performed computationally. Our experiments show that this analysis is accurate. The results of this study are useful for the design of high-performance thin film cells for wireless power transfer.

Wearable visual devices have many emerging applications in human health monitoring, such as the study of food intake and physical activities. However, these devices produce large amounts of data which must be processed efficiently and effectively. In this paper, we present a video- based health monitoring system for physical activity studies. We utilize an efficient signal extraction method to process and reduce the effects of noises on field-acquired image data. We further develop a maximum likelihood estimator to estimate the walking speed and classify activity patterns (walking or running) in real-time under both indoor and outdoor environments. We demonstrate the feasibility of the proposed method using three different test video sequences.

In this paper, we propose an approach for learning appearance models of moving objects directly from compressed video. The appearance of a moving object changes dynamically in video due to varying object poses, lighting conditions, and partial occlusions. Efficiently mining the appearance models of objects is a crucial and challenging technology to support content-based video coding, clustering, indexing, and retrieval at the object level. The proposed approach learns the appearance models of moving objects in the spatial-temporal dimension of video data by taking advantage of the MPEG video compression format. It detects a moving object and recovers the trajectory of each macroblock covered by the object using the motion vector present in the compressed stream. The appearances are then reconstructed in the DCT domain along the object's trajectory, and modeled as a mixture of Gaussians (MoG) using DCT coefficients. We prove that, under certain assumptions, the MoG model learned from the DCT domain can achieve pixel-level accuracy when transformed back to the spatial domain, and has a better band-selectivity compared to the MoG model learned in the spatial domain. We finally cluster the MoG models to merge the appearance models of the same object together for object-level content analysis.

Currently, a significant obstacle in identifying the most important factors that contribute to obesity is the lack of appropriate tools to evaluate food consumption and physical activity on a daily basis. This paper presents a novel video-based approach to estimating the energy intake and expenditure of individuals who are over-weight or obese. We propose to utilize a miniature video camera to chronically record data surrounding an individual's daily activity. From the recorded data, energy intake and expenditure are extracted objectively using signal processing algorithms. We discuss two algorithms which are both featured with low computational complexity, suitable for processing large sets of field-acquired video data. These algorithms estimate: (i) energy intake (caloric content) by determining the physical dimensions of the foods consumed; and (ii) energy expenditure by evaluating the extent of the subject's physical activity. We also report initial experimental results to demonstrate the effectiveness of the new approach.

Patient monitoring via video and physiological data recording can now be performed outside hospitals. This procedure, usually performed in a prolonged manner, generates a considerable amount of data, which calls for efficient ways for archiving and transmission. In this work, we present a specialized system to code the video and the physiological data recorded from a patient, aiming at a reduced bandwidth requirement compared to the conventional methods. We've developed an object-based approach to coding the monitoring video. By applying two change detection methods, we decompose a video frame into three video object planes (VOPs) representing the background, the stationary foreground and the moving foreground. These VOPs are coded at different frame rates, leading to a reduced overall bit rate. For coding the physiological data (using electroencephalogram, i.e. EEG, as an example), we present an effective solution by using a combination of the lifting scheme and the SPIHT algorithm. This approach is featured with a wavelet-quantization algorithm that enables a scalable transmission. The feasibility of this proposed system is demonstrated by our experimental results

Volume conduction communication is an efficient means of transmitting signals within the human body. The paper introduces a multiplexing and modulation method which minimizes circuit complexity, space, and power consumption.

Neurophysical interoperative monitoring (OIM) video is an important tool in biomedical applications and telemedicine and it is essential to maintain high fidelity of the frames even in low bit-rate environment. Toward this end, we studied a spatio-temporal 3D multiwavelet-based OIM video compression system that is capable of high quality output even under the demands of high compression ratios

Change detection is a useful tool to identify content changes in images. Its significance is found in the applications of video surveillance, editing, coding and analysis. In this work, we present another application of change detection to patient video monitoring, based on a novel approach in the discrete cosine transform (DCT) domain. This approach is built upon a statistical test on the difference between DCT-blocks, where the noise disturbances and illumination variations are excluded from the meaningful changes. Our experimental results show that the proposed method is robust in detecting content changes and may be used to facilitate high-level segmentation tasks

Transmitting high-quality neurophysiology video via the Internet is a challenging problem in telemedicine. We propose a novel video data processing and compression method based on the discrete wavelet transform (DWT). The DWT is utilized to decompose video frames into subframes, different bandwidth budgets are assigned to important and less important regions in each subframe, and several video encoders are employed in parallel to accelerate computation. When compared with the general-purpose video compression methods, our method allows scalability on real-time network bandwidth allocation, and offers higher video quality within the critical field of neurosurgery

This paper presents a study of applying an image superresolution method to enhance spatial resolution of MRI heart images from a temporal sequence. The particular method is a wavelet-based projection-onto-convex-set superresolution reconstruction. It makes use of the non-stationary effect in the successive images in the sequence to extract some detail information for image reconstruction at a higher spatial resolution. Our experimental result shows better visual quality than that obtained by using the cubic-spline interpolation.

This paper presents a novel method for objective assessment of speech quality based on one-step strategy using a feedfoward neutral network. Currently, almost all the existing methods for this assessment can be regarded as a two-step strategy, requiring a distortion computation and a mapping from the average distortion value to the mean opinion score (MOS). Our new method combines these two steps by means of a neural network which can incorporate the perception properties of the human auditory system and provide an MOS estimate directly. Our theoretical analysis and experimental results suggest that this method of MOS estimate significantly overperforms the traditional methods. The correlation coefficient between the subjective test score and objective MOS estimate can reach up to about 0.95.

Most real world signals consist of stronger low frequency components than high frequency ones. However, in many cases, high frequency components with small amplitude values represent the components of interest. Although time-frequency distributions (TFDs) have been commonly utilized to analyze such signals, the masking effect of large amplitude components, which severely reduce the sensitivity of TFDs, has been a significant problem. In this work we present a new method to pre-condition the input data using wavelet analysis and synthesis procedures before computing the TFD. This method attenuates interfering components in the TFD so that the masking effect is greatly reduced. EEG data are used to demonstrate the approach.

Multichannel electroencephalograms (EEGs) are processed using time-frequency (TF) analysis and synthesis techniques to geometrically localize neuroelectric generators of specific activity contained within the observed EEG. The TF domain techniques are utilized to separate the signals of interest from the remainder of the EEG, by allowing the definition of regions of interest which contain the signals for which we desire to localize the underlying neuronal generators. This approach essentially introduces a filtering technique which allows the distortionless separation of the signals of interest from all other components recorded. The source of the functional activity in the brain is estimated and mapped numerically by a least-squares approach. We have applied these techniques to identify the anatomical location of the sleep spindle, a component of the EEG observed during sleep, which is of importance in understanding the generation of sleep and sleep patterns.

The computational issues for symmetric wavelet transforms are investigated. We present a novel frequency-domain algorithm using the discrete cosine transforms (DCTs) and discrete sine transforms (DSTs). A high efficiency is achieved when this algorithm is applied to signals of finite duration, especially images.

A new approach is presented for time-series modeling and prediction using recurrent neural networks (RRNs) and a discrete wavelet transform (DWT). A specific DWT, based on the cubic spline wavelet, produces a set of wavelet coefficients from coarse to fine scale levels. The RNN has its current output fed back to its input nodes, forming a nonlinear autoregressive model for predicting future wavelet coefficients. A predicted trend signal is obtained by constructing the interpolation function from the predicted wavelet coefficients at the coarsest scale level, V/sub 0/. This method has been applied to intracranial pressure data collected from head trauma patients in the intensive care unit. The method has been shown to be more efficient than one which uses raw data to train the RNN.

In this paper the wavelet-based edge detection problem is investigated. Symmetrical wavelets of infinite support are studied which possess many desirable properties for edge localization in images. A design method for the digital filters that compute the wavelet transform is presented. These filters provide the minimum length under a given truncation error specification. Our method leads to an improved result for edge detection and a reduced computational cost.

A dynamic learning rate for back-propagation training of artificial neural networks is proposed as a weighted average of direction cosines of the incremental weight vectors of the current and previous steps. Experiments on training an EEG-based sleep state pattern recognition scheme have demonstrated its improved performance.>

A synchronously averaged signal, acquired in real time using an A/D converter, a package of service routines and a computer, can be corrupted if precautions are not taken to minimize imprecise timing (jitter). The effects of jitter on the recovery of a signal from noise by averaging is discussed. Two cases are examined. In the first case, the jitter is the result of frequency mismatch between the acquisition repetition frequency and the signal frequency. In the second case, the jitter is due to a random phase shift present in each of the acquired waveforms. Both cases result in a reduction in magnitude and a variation in latency. A detailed analysis is given deriving several statistical descriptors. It is established that these descriptors are closely related to the characteristic function of jitter. Special cases where jitter obeys the Gaussian and uniform distributions are examined. A simple technique for determining the mean and standard deviation of the jitter is also given by example.>

A study on applying the discrete pseudo-Wigner distribution to classification of the plasma cortisol short-time series of two disease categories is presented. The autocomponent selection in the Wigner distribution has been studied in regard to feature selection for the pattern recognition. Three energy density features are selected along with the mean level of each time series. These features allow a simple linear classification of the available data. The results show a great potential for using Wigner spectra for feature extraction.>

The authors introduce an autocomponent selection (ACS) algorithm to remove the cross-components produced by the discrete pseudo-Wigner distribution (DPWD). The ACS treats the DPWD as an image with polarity. This image is processed with an averaging filter to eliminate negative values. The result is then compared with a preprocessed DPWD to classify each image pixel. This approach yields a subset of the DPWD free of redundancies. Unlike traditional smoothing technique, this algorithm does not reduce the time-frequency resolution.>