Current methods of evaluating essential tremor (ET) either rely on subjective ratings or use limited tremor metrics (i.e., severity/amplitude and frequency). In this study, we explored performance metrics from Fitts’ law tasks that replicate and expand existing tremor metrics, to enable low-cost, home-based tremor quantification and analyze the cursor movements of individuals using a 3D mouse while performing a collection of drawing tasks. We analyzed the 3D mouse cursor movements of 11 patients with ET and three controls, on three computer-based tasks—a spiral navigation (SPN) task, a rectangular track navigation (RTN) task, and multi-directional tapping/clicking (MDT)—with several performance metrics (i.e., outside area (OA), throughput (TP in Fitts’ law), path efficiency (PE), and completion time (CT). Using an accelerometer and scores from the Essential Tremor Rating Assessment Scale (TETRAS), we correlated the proposed performance metrics with the baseline tremor metrics and found that the OA of the SPN and RTN tasks were strongly correlated with baseline tremor severity (R2= 0.57, and R2= 0.83). We also found that the TP in the MDT tasks were strongly correlated with tremor frequency (R2= 0.70). In addition, as the OA of the SPN and RTN tasks was correlated with tremor severity and frequency, it may represent an independent metric that increases the dimensionality of the characterization of an individual's tremor. Thus, this pilot study of the analysis of those with ET-associated tremor performing Fitts’ law tasks demonstrates the feasibility of introducing a new tremor metric that can be expanded for repeatable multi-dimensional data analyses.

We propose Dual Attention Networks (DANs) which jointly leverage visual and textual attention mechanisms to capture fine-grained interplay between vision and language. DANs attend to specific regions in images and words in text through multiple steps and gather essential information from both modalities. Based on this framework, we introduce two types of DANs for multimodal reasoning and matching, respectively. The reasoning model allows visual and textual attentions to steer each other during collaborative inference, which is useful for tasks such as Visual Question Answering (VQA). In addition, the matching model exploits the two attention mechanisms to estimate the similarity between images and sentences by focusing on their shared semantics. Our extensive experiments validate the effectiveness of DANs in combining vision and language, achieving the state-of-the-art performance on public benchmarks for VQA and image-text matching.

Precise item categorization is a key issue in e-commerce domains. However, it still remains a challenging problem due to data size, category skewness, and noisy metadata. Here, we demonstrate a successful report on a deep learning-based item categorization method, i.e., deep categorization network (DeepCN), in an e-commerce website. DeepCN is an end-to-end model using multiple recurrent neural networks (RNNs) dedicated to metadata attributes for generating features from text metadata and fully connected layers for classifying item categories from the generated features. The categorization errors are propagated back through the fully connected layers to the RNNs for weight update in the learning process. This deep learning-based approach allows diverse attributes to be integrated into a common representation, thus overcoming sparsity and scalability problems. We evaluate DeepCN on large-scale real-world data including more than 94 million items with approximately 4,100 leaf categories from a Korean e-commerce website. Experiment results show our method improves the categorization accuracy compared to the model using single RNN as well as a standard classification model using unigram-based bag-of-words. Furthermore, we investigate how much the model parameters and the used attributes influence categorization performances.

Deep neural networks continue to advance the state-of-the-art of image recognition tasks with various methods. However, applications of these methods to multimodality remain limited. We present Multimodal Residual Networks (MRN) for the multimodal residual learning of visual question-answering, which extends the idea of the deep residual learning. Unlike the deep residual learning, MRN effectively learns the joint representation from visual and language information. The main idea is to use element-wise multiplication for the joint residual mappings exploiting the residual learning of the attentional models in recent studies. Various alternative models introduced by multimodality are explored based on our study. We achieve the state-of-the-art results on the Visual QA dataset for both Open-Ended and Multiple-Choice tasks. Moreover, we introduce a novel method to visualize the attention effect of the joint representations for each learning block using back-propagation algorithm, even though the visual features are collapsed without spatial information.

Tongue drive system (TDS) is a novel tongue-operated assistive technology (AT) for the mobility impaired, to empower them to access computers and drive powered wheelchairs (PWC) using their free voluntary tongue motion. We have evaluated the TDS performance in five sessions over 5-8 weeks to study the learning process in different tasks of computer access and PWC navigation on nine able-bodied subjects who already had tongue piercing and used our magnetic tongue studs throughout the trial. Computer access tasks included on-screen maze navigation and issuing random commands to measure the TDS information transfer rate. PWC navigation included driving through a ~50-m obstacle course using three control strategies. Some of the qualitative aspects of using the TDS were also evaluated based on the two Likert scale questionnaires, one of which was short (eight questions) and asked at the end of each session and the other one (46 questions) at the end of the trial. Included in this study was also a task to measure the tongue fatigue as a result of using the TDS continuously for a few hours. All performance measures showed significant improvement from the first to the second session as well as further gradual improvements throughout the rest of the sessions, suggesting a rapid learning process.

In a deserted world, Captain Banana flies to the Banana swinging boat to operate it. The impatient passengers of the attraction, the so-called "Littles" force him to drive the boat faster and faster! Eventually, as the system overloads, the banana boat crashes and the littles bounce out the ball and fall into the nearby oasis. Captain Banana tries to avoid it but he ultimately fails to stop one of the littles to fall into the oasis, starting a process that was not meant to happen, but there is no way back... The animation presents a metaphor of the human conception process and the issues and concerns related to it. Captain Banana is the inner self of a normal man, he is sure of himself and confident of his capabilities but there is things that even a carefree super-hero cannot stop. With this short animation we wish to convey the message of how important is to "think healthy and safely" to avoid undesired situations.

Path expressions have been accepted for concisely manipulating the nested structures in complex object-oriented query expressions. However, previous visual query languages hardly represent such query expressions in a concise and intuitive way partly due to improper visual representation of path expressions and partly due to lack of well-defined system and semantics of languages. In this paper, we present visual modeling of path expressions in a visual object-oriented database query language called Visual Object-Oriented Query Language (VOQL) which has excellent expressive pourer for sets, simple and intuitive syntax, and well-defined semantics. This is enabled by explicitly specifying the semantics of multi-valued path expressions based on the visual notation capable of representing set relationships in addition to functional relationships. The basic visual constructs called blobs and nested blobs denote sets of objects that path expressions represent while the constructs called binding edges and flattening edges visually simulate the notions of variable binding and dot functions in path expressions respectively. Based on the constructs, the grammar of VOQL defines the syntactic components while the semantics of query expressions are provided by syntax-directed translation to the counterparts in the extended relational calculus. Also, the visual constructs allow modeling of restricted universal quantification with a visual scoping box and effectively represent nested quantification and recursive queries without semantic ambiguities.