This article explores video analysis methods for monitoring eating behaviors, a critical factor in approximately 70% of global deaths due to illnesses like cancer, diabetes, and heart disease. Automated monitoring quantifies aspects such as meal duration, food types, and intake gestures (bite and drink gestures). Previous deep-learning methods segment videos into short clips (e.g., 16 frames at 8 Hz) for analysis, but this approach overlooks common meal-length patterns in gesture distribution across different individuals and sessions, which can enhance detection accuracy. Our study introduces a novel pipeline that analyzes the entire meal context (5-40 minutes). We propose a framework allowing a global detector to learn meal-length patterns with manageable computational demands. Additionally, we introduced a new augmentation technique to generate hundreds of meal-length feature samples per video, facilitating effective training of a global detector with limited video availability. Experimental results on two datasets (Clemson Cafeteria and EatSense) demonstrate that our pipeline significantly enhances the performance of state-of-the-art window-based networks, particularly in reducing false positives in gesture detection. On the Clemson Cafeteria dataset of 486 meal videos (the largest dataset to date), our method achieves F1 scores of 0.93 for bite gestures and 0.88 for drink gestures, substantially outperforming existing methodologies.

This paper presents new methods to detect eating from wrist motion. Our main novelty is that we analyze a full day of wrist motion data as a single sample so that the detection of eating occurrences can benefit from diurnal context. We develop a two-stage framework to facilitate a feasible full-day analysis. The first-stage model calculates local probabilities of eating$P(E_{w})$within windows of data, and the second-stage model calculates enhanced probabilities of eating$P(E_{d})$by treating all$P(E_{w})$within a single day as one sample. The framework also incorporates an augmentation technique, which involves the iterative retraining of the first-stage model. This allows us to generate a sufficient number of day-length samples from datasets of limited size. We test our methods on the publicly available Clemson All-Day (CAD) dataset and FreeFIC dataset, and find that the inclusion of day-length analysis substantially improves accuracy in detecting eating episodes. We also benchmark our results against several state-of-the-art methods. Our approach achieved an eating episode true positive rate (TPR) of 89% with 1.4 false positives per true positive (FP/TP), and a time weighted accuracy of 84%, which are the highest accuracies reported on the CAD dataset. Our results show that the daily pattern classifier substantially improves meal detections and in particular reduces transient false detections that tend to occur when relying on shorter windows to look for individual ingestion or consumption events.

Intake gesture detection plays an important role in dietary monitoring systems to help individuals objectively quantify their eating behavior. Intake gestures include motions related to consumption, such as ingestion of a bite (food) and drink (liquid). Video provides an automatic and non-contact way to detect gestures. Previous studies have evaluated video-based detectors in laboratories where participants were invited to consume meals containing a limited number of food and beverage choices. Because of this, previous datasets have limited variability in food and beverage appearances, and types of intake tools and gestures. Cameras were positioned conspicuously close to participants. To address these limitations, we describe a new video dataset that was collected by recording the meals of 264 participants in a real restaurant. Participants consumed 374 different types of foods and beverages. The camera was installed on the ceiling of the restaurant to minimize the effect of an experimental observer. In this study, we present the details of collecting and labeling the data for video-based research. To explore the feasibility of detecting intake gestures using our dataset and establish baseline performance, we implemented three state-of-art models, namely CNN-LSTM, SlowFast and X3D-L. Results indicate that the best F1 scores were 0.899 and 0.778 on bite and drink gesture detection respectively.

Many works in the field of automated dietary monitoring (ADM) have analyzed small data sets consisting of <; 10 subjects and <; 20 meals. This is often the first step in researching new sensors or body positions for detecting consumption. Metrics tend to focus on within-meal accuracy by quantifying physiological event detection (bites, chews, swallows). As analysis shifts to larger datasets containing many days of data from everyday life and researchers build methods that can be used in everyday life, it becomes equally important to quantify the accuracy of how many meals are detected. In small data sets most meals can be detected at least partially. In larger datasets, some meals are missed and false positives occur. In this work we discuss the pros and cons of time-based metrics and episode level metrics. We demonstrate how class imbalance affects some of the commonly used time metrics, and discuss why episode level metrics need to be reported as the field transitions from small data sets to big data sets.

This paper describes a novel approach of segmenting and classifying eating gestures from wrist motion using a deep learning neural network. It is inspired by the approach of fully-convolutional neural networks in the task of image segmentation. Our idea is to segment 1D gestures the same way 2D image regions are segmented, by treating each inertial measurement unit (IMU) datum like a pixel. The novelty of our approach lies in training a neural network to recognize data points that describe an eating gesture just like it would be trained to recognize pixels describing an image region. The data for this research is known as the Clemson Cafeteria Dataset. It was collected from 276 participants that ate an unscripted meal at the Harcombe Dining Hall at Clemson University. Each meal consisted of 1 - 4 courses, and 488 such recordings were used for the experiments described in this paper. Sensor readings consist of measurements taken by an accelerometer (x, y, z) and a gyroscope (yaw, pitch, roll). A total of 51,614 unique gestures associated with different activities commonly seen during a meal were identified by 18 trained raters. Our neural network classifier recognized an average of 79.7% of ‘bite’ and 84.7% of ‘drink’ gestures correctly per meal. Overall 77.7% of all gestures were recognized correctly on average per meal. This indicates that a deep learning model can successfully be used to segment eating gestures from a time series recording of IMU data using a technique similar to pixel segmentation within an image.

This article considers detecting eating in free-living humans by tracking wrist motion. We are specifically interested in the effect of secondary activities that people conduct while simultaneously eating, such as walking, watching television, or working. These secondary activities cause wrist motions that obfuscate those associated with eating, increasing the difficulty of detecting periods of eating. We collected a large dataset of 4,680 hours of wrist motion from 351 participants during free living. Participants reported secondary activities in 72% of meals. Analysis of wrist motion data revealed that the wrist was resting 12.8% of the time during self-reported meals compared to only 6.8% of the time in a cafeteria dataset, whereas walking motion was found 5.5% of the time during meals in free living compared to 0% in a cafeteria. Augmenting an eating detection classifier to include walking and resting detection improved accuracy from 74% to 77% on our free-living dataset ( t [353] = 7.86, p < 0.001). Although eating detection could be improved using more sophisticated machine learning methods or sensor modalities, all approaches would be affected by secondary activities, as they affect the labeling of data itself. Our work suggests that future work should collect detailed ground truth on secondary activities being conducted during eating, as these activities could hold insights into when an eating activity starts or stops in the absence of video-based ground truth.

Accelerometers have gained popularity in biomedical and m-health applications such as actigraphy or automated dietary monitoring due to their ease of use and their ability to characterize motion. These sensors report raw acceleration from which gravity and linear acceleration must be separated, with commercial packages reporting raw acceleration, linear acceleration or both. New researchers to the field may often be confused when to use raw acceleration or linear acceleration, especially given the susceptibility of linear acceleration to noise, and the lack of published distributions of these signals. This paper provides a short tutorial on obtaining linear acceleration estimates. Using these methods we analyze a large dataset containing 4,680 hours of wrist tracking data, the largest such dataset known to us. We learn the range of wrist motion accelerations, and quantify the expected noise in the linear acceleration signal. We explain the sources of this noise, and a filtering technique to mitigate it. For the first time, we report the range of wrist acceleration values observed during free-living, and quantify the expected range of noise in this wrist acceleration. We show that while previous work has reported average accelerations at the feet and body ranging from 0-15g during spots-like activities like walking, running or jumping, wrist acceleration in free-living subjects during daily activities is often much lower, and ranges from 0-0.2g. We show that noise in linear acceleration can range from 0-0.06g, an overlap of 70%. This suggests that in applications where the wrist acceleration is in this range of noise, linear acceleration may not provide useful features, and researchers should only rely on raw acceleration instead.

This paper evaluates the use of a sensor mounted on a finger to detect bites during food consumption. Previous works have demonstrated the capability to detect bites by tracking wrist motion. The key motion is the roll of the wrist that occurs between picking up food and delivering it to the mouth. We hypothesize that the roll axes of the wrist, hand, and finger are congruent, and that therefore the roll motion measured at the wrist can be similarly measured at the finger. We collected data for 10 meals of subjects wearing sensors on both the wrist and finger. Results indicate that the bite counting algorithm obtains similar accuracy at both locations. The practical implication is that an eating activity monitor could be embedded in the form of a ring worn on the finger, and that this form factor could work as well as a watch-like device.

This work describes a new dataset to improve pedometer evaluation. Prior evaluation techniques focus on regular gaits using laboratory assessment to simplify the manual counting of actual steps. Our goal is to analyze pedometer algorithms under more natural conditions that occur during daily living where gaits are frequently changing or remain regular for only brief periods of time. We video recorded 30 participants performing 3 activities: walking around a track, walking through a building, and moving around a room. Walking around a track uses a regular, consistent gait, and represents the traditional approach to pedometer evaluation. Walking through a building and around a room are activities that include varying amounts of pauses and gait changes, and represent a wider variety of normal daily activities. The ground truth time of each step was manually marked in the accelerometer signals by observing the videos. Collectively 60,853 steps were recorded and annotated. A subclass of steps called shifts were identified as those occurring at the beginning and end of regular strides, during gait changes, and during pivots changing the direction of motion. While shifts comprised only .03% of steps in the regular stride activity, they comprised 10-25% of steps in the semi-regular and unstructured activities. We believe these motions should be identified separately, as they provide different accelerometer signals, and likely result in different amounts of energy expenditure. The proposed dataset will be the first to specifically allow for pedometer algorithms to be evaluated on unstructured gaits that more closely model natural activities.

The universal eating monitor (UEM) is a table-embedded scale used to measure grams consumed over time while a person eats. It has been used in laboratory settings to test the effects of anorectic drugs and behavior manipulations such as slowing eating, and to study relationships between demographics and body weight. However, its use requires restricted conditions on the foods consumed and behaviors allowed during eating in order to simplify analysis of the scale data. Individual bites can only be measured when the only interaction with the scale is to carefully remove a single bite of food, consume it fully, and wait a minimum amount of time before the next bite. Other interactions are prohibited such as stirring and manipulating foods, retrieving or placing napkins or utensils on the scale, and in general anything that would change the scale weight that was not related to the consumption of an individual bite. This paper describes a new algorithm that can detect and measure the weight or individual bites consumed during unrestricted eating. The algorithm works by identifying time periods when the scale weight is stable, and then, analyzing the surrounding weight changes. The series of preceding and succeeding weight changes is compared against patterns for single food bites, food mass bites, and drink bites to determine if a scale interaction is due to a bite or some other activity. The method was tested on 271 subjects, each eating a single meal in a cafeteria setting. A total of 24 101 bites were manually annotated in synchronized videos to establish ground truth as to the true, false, and missed detections of bites. Our algorithm correctly detected and weighed approximately 39% of bites with approximately one false positive (FP) per ten actual bites. The improvement compared to the UEM is approximately three times the number of true detections and a 90% reduction in the number of FPs. Finally, an analysis of bites that could not be weighed compared to those that could be weighed revealed no statistically significant difference in average weight. These results suggest that our algorithm could be used to conduct studies using a table scale outside of laboratory or clinical settings and with unrestricted eating behaviors.

This paper describes a study to test the accuracy of a method that tracks wrist motion during eating to detect and count bites. The purpose was to assess its accuracy across demographic (age, gender, and ethnicity) and bite (utensil, container, hand used, and food type) variables. Data were collected in a cafeteria under normal eating conditions. A total of 271 participants ate a single meal while wearing a watch-like device to track their wrist motion. A video was simultaneously recorded of each participant and subsequently reviewed to determine the ground truth times of bites. Bite times were operationally defined as the moment when food or beverage was placed into the mouth. Food and beverage choices were not scripted or restricted. Participants were seated in groups of 2-4 and were encouraged to eat naturally. A total of 24 088 bites of 374 different food and beverage items were consumed. Overall the method for automatically detecting bites had a sensitivity of 75% with a positive predictive value of 89%. A range of 62-86% sensitivity was found across demographic variables with slower eating rates trending toward higher sensitivity. Variations in sensitivity due to food type showed a modest correlation with the total wrist motion during the bite, possibly due to an increase in head-toward-plate motion and decrease in hand-toward-mouth motion for some food types. Overall, the findings provide the largest evidence to date that the method produces a reliable automated measure of intake during unrestricted eating.

This paper considers the problem of recognizing eating gestures by tracking wrist motion. Eating gestures are activities commonly undertaken during the consumption of a meal, such as sipping a drink of liquid or using utensils to cut food. Each of these gestures causes a pattern of wrist motion that can be tracked to automatically identify the activity. Previous works have studied this problem at the level of a single gesture. In this paper, we demonstrate that individual gestures have sequential dependence. To study this, three types of classifiers were built: 1) a K-nearest neighbor classifier which uses no sequential context, 2) a hidden Markov model (HMM) which captures the sequential context of subgesture motions, and 3) HMMs that model intergesture sequential dependencies. We built first-order to sixth-order HMMs to evaluate the usefulness of increasing amounts of sequential dependence to aid recognition. On a dataset of 25 meals, we found that the baseline accuracies for the KNN and the subgesture HMM classifiers were 75.8% and 84.3%, respectively. Using HMMs that model intergesture sequential dependencies, we were able to increase accuracy to up to 96.5%. These results demonstrate that sequential dependencies exist between eating gestures and that they can be exploited to improve recognition accuracy.

This paper is motivated by the growing prevalence of obesity, a health problem affecting over 500 million people. Measurements of energy intake are commonly used for the study and treatment of obesity. However, the most widely used tools rely upon self-report and require a considerable manual effort, leading to underreporting of consumption, noncompliance, and discontinued use over the long term. The purpose of this paper is to describe a new method that uses a watch-like configuration of sensors to continuously track wrist motion throughout the day and automatically detect periods of eating. Our method uses the novel idea that meals tend to be preceded and succeeded by the periods of vigorous wrist motion. We describe an algorithm that segments and classifies such periods as eating or noneating activities. We also evaluate our method on a large dataset (43 subjects, 449 total h of data, containing 116 periods of eating) collected during free-living. Our results show an accuracy of 81% for detecting eating at 1-s resolution in comparison to manually marked event logs of periods eating. These results indicate that vigorous wrist motion is a useful indicator for identifying the boundaries of eating activities, and that our method should prove useful in the continued development of body-worn sensor tools for monitoring energy intake.

We introduce methods for detecting in real-time information concerning bites taken during a meal. Our methods use an orientation sensor placed on the wrist of a user, and analyze the rolling motion of the wrist in order to detect a pattern related to biting behavior. We have built a prototype bite detector device based upon our methods. The device can count the total number of bites the user has taken, and provide the bites-taken rate (bites per minute) of the user. Experiments have been conducted to determine its accuracy. Ten subjects ate a meal of their choice, using utensils (or fingers) of their choice. Video was recorded of subjects eating, and synchronized with our device, in order to evaluate its performance. The sensitivity of the device was found to be 91%. Our methods could find use in a number of applications, including helping a user with obesity, eating disorders or eating rate problems.

Allowing dynamic motions for payloads within a workcell is a fundamentally novel idea for practical industrial robots. The ability to handle payloads moving in semi-structured ways would significantly increase the potential markets for industrial robotics. However, due to the inherent time delay between complex sensing and manipulation, the manipulation fails in some circumstances. Therefore, in this paper, we propose a generic method to model the dynamic intercept and manipulation capability of vision based industrial robot systems. In order to verify the method, we present experiments using our industrial workcell prototype to dynamically intercept and manipulate semi-randomly moving objects. We conduct experiments over 1000 runs with two different kinds of dynamic tasks, "scoop" and "trap." The experimental results validate our theory

A computer system today receives no data regarding the physiological state of the user, but there are many cases where these data could be useful. For example, as the user becomes bored or lethargic, the system could raise the workload or audiovisual feedback to stimulate arousal. As the user becomes tense or strained, the system could lighten the workload or simplify the feedback to lessen arousal. This type of physiological-based closed-loop feedback could be applied in a number of scenarios, such as training, stressful repetitive work (e.g., air traffic control), and military operations. In this article, the authors describe a computing system that produces a real-time cardiac-based measure of arousal. The measure is based on changes in respiratory sinus arrhythmia, an established measure of vagal activity. They describe the measure, the measure's potential and limits, the computing system, and the new directions for physiological monitoring research made possible by such a system.

Visual sensing for robotics has been around for decades, but our understanding of a timing model remains crude. By timing model, we refer to the delays (processing lag and motion lag) between "reality" (when a part is sensed), through data processing (the processing of image data to determine part position and orientation), through control (the computation and initiation of robot motion), through "arrival" (when the robot reaches the commanded goal). In this study, we introduce a timing model where sensing and control operate asynchronously. We apply this model to a robotic workcell consisting of a Sta/spl uml/ubli RX-130 industrial robot manipulator, a network of six cameras for sensing, and an off-the-shelf Adept MV-19 controller. We present experiments to demonstrate how the model can be applied.

Visual sensing for robotics has been around for decades, but our understanding of a timing model remains crude. By timing model, we refer to the delays (processing lag and motion lag) between "reality" (when a part is sensed), through data processing (the processing of image data to determine part position and orientation), through control (the computation and initiation of robot motion), through "arrival" (when the robot reaches "reality"). In this work we introduce a timing model where sensing and control operate asynchronously. We apply this model to a robotic workcell consisting of a Staubli RX-130 industrial robot manipulator, a network of six cameras for sensing, and an off-the-shelf Adept MV-19 controller. We demonstrate some experiments to show how the model is applied.

We describe an automated method to locate the optic nerve in images of the ocular fundus. Our method uses a novel algorithm we call fuzzy convergence to determine the origination of the blood vessel network. We evaluate our method using 31 images of healthy retinas and 50 images of diseased retinas, containing such diverse symptoms as tortuous vessels, choroidal neovascularization, and hemorrhages that completely obscure the actual nerve. On this difficult data set, our method achieved 89% correct detection. We also compare our method against three simpler methods, demonstrating the performance improvement. All our images and data are freely available for other researchers to use in evaluating related methods.

In this paper we present a method to compute the egomotion of a range camera using the space envelope. The space envelope is a geometric model that provides more information than a simple segmentation for correspondences and motion estimation. We describe a novel variation of the maximal matching algorithm that matches surface normals to find correspondences. These correspondences are used to compute rotation and translation estimates of the egomotion. We demonstrate our methods on two image sequences containing 70 images. We also discuss the cases where our methods fail, and additional possible methods for exploiting the space envelope.

As manipulators, robot arms are woefully under-utilized in modern industrial workcells. The norm involves little or no sensing, the grasp of a single known object, and the execution of a single geometric motion. In contrast, the popular ideal of a visually guided arm (as for a human) should be able to track and grab relatively arbitrary objects, under relatively arbitrary motions, across a comfortable range of velocities. Our ongoing research aims at extending these capabilities in a modern industrial workcell. We outline a framework in which these capabilities might be measured, and demonstrate experimentally the capabilities of our current prototype.

We present a novel approach to the workcell design for enhanced performance of industrial robot manipulators, in which the workcell features a specially integrated sensor network. In contrast to traditional robot workcells, in which the role of environmental sensors (if present at all) is relatively inflexible and tightly focused on a specific task, the sensor network performs real-time dynamic sensing of the entire workcell. The system makes use of "off the shelf" components, and integrates them with the existing industrial robot controller, to allow the manipulator to perform more dynamic operations in a less structured workcell. Details of the novel sensor networked workcell and experiments with an industrial robot are presented.

The dominant architecture for mobile robot perception uses sensors on-board the robot, providing a first-person perspective on the environment. We demonstrate a novel mobile robot architecture that uses an environment-based sensor network, which provides third-person perception. The idea is that a mobile robot working in the area tunes in to broadcasts from the video camera network (or in this case from an environment-based computer processing the video frames) to receive sensor data. This distributed sensing configuration offers several advantages over on-board sensing.

The dominant architecture for mobile robot perception uses sensors on-board the robot, providing only a first-person perspective on the environment. This work describes a novel mobile robot system that uses an environment-based sensor network, providing a powerful third-person perspective on the environment. In our previous work (1999), we described an algorithm that computes a real-time spatial-temporal occupancy map from the multiple video streams of the sensor network. We also described a novel path-planning algorithm based upon this system. In this work we describe a novel motion control loop that is based upon tracking the mobile robot in the occupancy map. Tracking in the fused perceptual space of the sensor network provides several advantages over tracking individually in a set of raw sensor spaces. We demonstrate a prototype system operating in several dynamic scenarios.

We describe an automated method to locate and outline blood vessels in images of the ocular fundus. Such a tool should prove useful to eye care specialists for purposes of patient screening, treatment evaluation, and clinical study. Our method differs from previously known methods in that it uses local and global vessel features cooperatively to segment the vessel network. We evaluate our method using hand-labeled ground truth segmentations of 20 images. A plot of the operating characteristic shows that our method reduces false positives by as much as 15 times over basic thresholding of a matched filter response (MFR), at up to a 75% true positive rate. For a baseline, we also compared the ground truth against a second hand-labeling, yielding a 90% true positive and a 4% false positive detection rate, on average. These numbers suggest there is still room for a 15% true positive rate improvement, with the same false positive rate, over our method. We are making all our images and hand labelings publicly available for interested researchers to use in evaluating related methods.

We describe an algorithm to fuse intensity data from multiple video cameras to create a spatial-temporal occupancy map. The camera layout is assumed to resemble a security video network. The occupancy map is a two-dimensional raster image, uniformly distributed in the floor-plane. Each map pixel contains a binary value, signifying whether the designated floorspace is empty or occupied. Our algorithm requires only one difference and one look-up table operation to determine each pixel's effect upon the map. This brevity of operations allows the spatial occupancy map to be temporally computed at real-time video rates. We demonstrate our algorithm operating in several dynamic scenarios.

This work considers the problem of discovering areas of convergence of line-like shapes in an image. The motivating application is to use the convergence of the blood vessel network to automatically locate the optic nerve in an ocular fundus image. A fuzzy segment model is proposed, based on a conjecture that line-like shapes only contribute to a perception of convergence in their near neighborhood. Using this model, a voting-type method is described to compute a convergence image, which can be searched for one absolute, or one or more relative, strongest points of convergence. Results are presented for twenty ocular fundus images, with a 65% success rate for finding the optic nerve.

The construction of a surface model from range data may be undertaken at any point in a continuum of scales that reflects the level of detail of the resulting model. This continuum relates the construction parameters to the scale of the model. We propose methods to dynamically reprocess range data at different scales. The construction result from a single scale is automatically evaluated, causing reconstruction at a different scale when user-defined criteria are not met. We demonstrate our methods in constructing a planar b-rep space envelope (a scene representation) for over 400 range images. The experiments demonstrate the ability to construct 100 percent valid models, with the scale of detail within specified requirements.

A methodology for evaluating range image segmentation algorithms is proposed. This methodology involves (1) a common set of 40 laser range finder images and 40 structured light scanner images that have manually specified ground truth and (2) a set of defined performance metrics for instances of correctly segmented, missed, and noise regions, over- and under-segmentation, and accuracy of the recovered geometry. A tool is used to objectively compare a machine generated segmentation against the specified ground truth. Four research groups have contributed to evaluate their own algorithm for segmenting a range image into planar patches.

Knowledge about required functionality of an object can be used as an effective representation for a generic object category (e.g. "chair", "cup", or "hammer"). This approach to object representation and recognition has recently become an active area of research. We explore a scenario in which a robot senses the environment to obtain an initial partial shape model of an object. If the information in this initial model is not sufficient to hypothesize a possible function for the object, then additional view(s) may be suggested. Once a possible function is hypothesized, a plan is formulated for interacting with the object to confirm that its material properties are compatible with the hypothesized function. The module for reasoning about partial shape models has been evaluated on over 200 shape models acquired from range images. The module for carrying out a function verification plan has been evaluated in a simulated environment using the ThingWorld (TW) system.

A new approach is presented for extracting an explicit 3D shape model from a single range image. One novel aspect is that the model represents both observed object surfaces, and surfaces which bound the volume of occluded space. Another novel aspect is that the approach does not require that the range image segmentation be perfect. The low-level segmentation may be such that the model-building process encounters topology versus geometry conflicts. The model-building process is designed to be "fail soft" in the face of such problems. The portion of the 3D model where a problem presents itself is "glued" together in a manner meant to minimize the disturbance in the 3D shape. The goal is to produce a valid boundary-representation which can be processed by higher-level routines. A third novel aspect of this work is that the implementation has been evaluated on over 200 real range images of polyhedral objects, with no operator intervention and all parameters held constant, and obtained a 97% success rate in creating valid b-reps.>

This paper describes a definition of the range image segmentation (of polyhedral scenes) problem, a data set to use in evaluation, a method for specifying ground truth, and a set of metrics to classify segmentation results against ground truths.>