This paper presents LAPA (Look Around and Pay Attention), a novel end-to-end transformer-based architecture for multi-camera point tracking that integrates appearancebased matching with geometric constraints. Traditional pipelines decouple detection, association, and tracking, leading to error propagation and temporal inconsistency in challenging scenarios. LAPA addresses these limitations by leveraging attention mechanisms to jointly reason across views and time, establishing soft correspondences through a cross-view attention mechanism enhanced with geometric priors. Instead of relying on classical triangulation, we construct 3D point representations via attention-weighted aggregation, inherently accommodating uncertainty and partial observations. Temporal consistency is further maintained through a transformer decoder that models long-range dependencies, preserving identities through extended occlusions. Extensive experiments on challenging datasets, including our newly created multi-camera (MC) versions of TAPVid-3D panoptic and PointOdyssey, demonstrate that our unified approach significantly outperforms existing methods, achieving 37.5% APD on TAPVid-3D-MC and 90.3% APD on PointOdyssey-MC, particularly excelling in scenarios with complex motions and occlusions. Code is available at https://github.com/ostadabbas/Look-Around-and-Pay-Attention-LAPA-.

In dynamic Neural Radiance Fields (NeRF) systems, state-of-the-art novel view synthesis methods often fail under significant viewpoint deviations, producing unstable and unrealistic renderings. To address this, we introduce Expanded Dynamic NeRF (ExpanDyNeRF), a monocular NeRF framework that leverages Gaussian splatting priors and a pseudo-ground-truth generation strategy to enable realistic synthesis under large-angle rotations. ExpanDyNeRF optimizes density and color features to improve scene reconstruction from challenging perspectives. We also present the Synthetic Dynamic Multiview (SynDM) dataset, which is the first synthetic multiview dataset for dynamic scenes with explicit side-view supervision, created using a custom GTA V-based rendering pipeline. Quantitative and qualitative results on SynDM and real-world datasets demonstrate that ExpanDyNeRF significantly outperforms existing dynamic NeRF methods in rendering fidelity under extreme viewpoint shifts. Further details are provided in the supplementary materials. Code is available at https://github.com/ostadabbas/ExpanDyNeRF.

The development of contactless respiration monitoring for infants could enable advances in the early detection and treatment of breathing irregularities, which are associated with neurodevelopmental impairments and conditions like sudden infant death syndrome (SIDS). But while respiration estimation for adults is supported by a robust ecosystem of computer vision algorithms and video datasets, only one small public video dataset with annotated respiration data for infant subjects exists, and there are no reproducible algorithms which are effective for infants. We introduce the annotated infant respiration dataset of 400 videos (AIR-400), contributing 275 new, carefully annotated videos from 10 recruited subjects to the public corpus. We develop the first reproducible pipelines for infant respiration estimation, based on infant-specific region-of-interest detection and spatiotemporal neural processing enhanced by optical flow inputs. We establish, through comprehensive experiments, the first reproducible benchmarks for the state-of-the-art in vision-based infant respiration estimation. We make our dataset, code repository, and trained models available for public use.

This paper introduces MOTE (MOre Than meets the Eye), a novel multi-object tracking (MOT) algorithm designed to address the challenges of tracking occluded objects. By integrating deformable detection transformers with a custom disocclusion matrix, MOTE significantly enhances the ability to track objects even when they are temporarily hidden from view. The algorithm leverages optical flow to generate features that are processed through a softmax splatting layer, which aids in the creation of a disocclusion matrix. This matrix plays a crucial role in maintaining track consistency by estimating the motion of occluded objects. MOTE's architecture includes modifications to the enhanced track embedding module (ETEM), which allows it to incorporate these advanced features into the track query layer embeddings. This integration ensures that the model not only tracks visible objects but also accurately predicts the trajectories of occluded ones, much like the human visual system. The proposed method is evaluated on multiple datasets, including MOT17, MOT20, and DanceTrack, where it achieves impressive tracking metrics--82.0 MOTA and 66.3 HOTA on the MOT17 dataset, 81.7 MOTA and 65.8 HOTA on the MOT20 dataset, and 93.2 MOTA and 74.2 HOTA on the DanceTrack dataset. Notably, MOTE excels in reducing identity switches and maintaining consistent tracking in complex real-world scenarios with frequent occlusions, outperforming existing state-of-the-art methods across all tested benchmarks.

Diffusion models have proven effective at generating high-quality images from learned distributions, but their application to the temporal domain, especially for driving scenarios, remains underexplored. Our work addresses key challenges in existing simulations, such as limited data quality, diversity, and high costs, by extending diffusion models to generate realistic long driving videos. We introduce the Dualconditioned Temporal Diffusion Model (DcTDM), an opensource method that incorporates dual conditioning to enforce temporal consistency by guiding frame transitions. Alongside DcTDM, we present DriveSceneDDM, a comprehensive driving video dataset featuring textual scene descriptions, dense depth maps, and canny edge data. We evaluate DcTDM using common video quality metrics, demonstrating its superior performance over other video diffusion models by producing long, temporally consistent driving videos up to 40s, achieving over 25% improvement in consistency and frame quality.

This paper introduces the dynamic robust adaptive graph-based tracker (DragonTrack), as a novel end-to-end framework for multi-person tracking (MPT) by integrating a detection transformer model for object detection and feature extraction with a graph convolutional network for re-identification. DragonTrack leverages encoded features from the transformer for precise subject matching and track maintenance, while the graphical component processes these features alongside geometric data to predict subsequent positions of tracked people. This methodology aims to enhance tracking accuracy and reliability, as evidenced by improvements in key metrics such as higher order tracking accuracy (HOTA) and multiple object tracking accuracy (MOTA). We quantitatively compare Drag-onTrack with state-of-the-art methods on MOT17, MOT20, and DanceTrack datasets, in which DragonTrack outperforms other methods. In challenging scenarios such as DanceTrack, DragonTrack achieves an impressive MOTA score of 93.4, significantly higher than the second-best SOTA method, ByteTrack, which achieves only 89.6. Similarly, on MOT17, DragonTrack scores 82.0 in MOTA, sur-passing the closest competitor with a score of 80.3. On MOT20, DragonTrack attains a HOTA score of 63.2, out-performing the next best method scoring 62.611The DragonTrack code is available at https://github.com/ostadabbas/DragonTrack. .

Estimating 3D human pose from 2D images in real world contexts remains a challenge, characterized by unique data constraints. Large general datasets of motion-captured 3D adult human poses paired with 2D images exist, but in many application settings, collection of further motion-captured data is impossible, precluding a straightforward fine-tuning approach to adaptation. We present a method for improving 3D pose estimation transfer learning to domains where there are only depth camera images available as supervision. Our heuristic weakly supervised 3D human pose (HW-HuP) estimation method learns partial pose priors from general 3D human pose datasets and employs weak supervision with depth data to guide learning in an optimization and regression cycle. We show that HW-HuP meaningfully improves upon state- of-the-art models in the adult in-bed setting, as well as on large scale public 3D human pose datasets, under comparable supervision conditions. Our model code and data are publicly available at https://github.com/ostadabbas/hw-hup. A significantly expanded version of this paper, with supplementary material, is available as a preprint on arXiv at https://arxiv.org/abs/2105.10996.

Assessing speech disorders in early childhood is challenging, and Speech-Language Pathologists (SLPs) play a key role in addressing such a challenge. However, tools that support speech assessment are often not child-friendly, and SLPs struggle to keep young patients engaged. To compensate, we introduce ArticuMotion, a child-friendly app that supports the assessment of speech disorders while engaging children in gamified experiences. We use participatory design to co-create ArticuMotion with SLPs and test the resulting product in a user study with nine preschool children. ArticuMotion has promise as a gamified assessment for motor speech disorders and shows a potential avenue for designing clinical tools that are useful while being child-friendly.

Recent advancements in object detection and human activity recognition have shown commendable progress, albeit with a predominant focus on adult-centric applications and datasets. This paper proposes a new vision-based, infant-focused hazard detection framework, CribNet, to assess threats to in-crib safety in the form of blanket occlusions and hazardous toys, as a step towards addressing the broad, critical problem of infant sleep safety. CribNet estimates hazards by considering the proximity and characteristics of detected objects around the infants. To evaluate the framework, we created the first publicly available crib hazard detection (CribHD) dataset, consisting of 1,620 images specific to infant-centric environments. These images present a wide range of real-world challenges, including clutter, occlusion, varied lighting conditions, with and without presence of infants in the images. We show that the framework performs with over 80% mean average precision (mAP) in segmenting toys and blankets and accurately assessing hazards, marking a new advancement in infant safety. CribNet and CribHD lay the foundation for future developments in in-crib hazard detection and infant sleep safety11The code and our data are publicly available at https://github.com/ostadabbas/CribNet.

We apply computer vision pose estimation techniques developed expressly for the data-scarce infant domain to the study of torticollis, a common condition in infants for which early identification and treatment is critical. Specifically, we use a combination of facial landmark and body joint estimation techniques designed for infants to estimate a range of geometric measures pertaining to face and upper body symmetry, drawn from an array of sources in the physical therapy and ophthal-mology research literature in torticollis. We gauge performance with a range of metrics and show that the estimates of most these geometric measures are successful, yielding strong to very strong Spearman's$p$correlation with ground truth values. Furthermore, we show that these estimates, derived from pose estimation neural networks designed for the infant domain, cleanly outperform estimates derived from more widely known networks designed for the adult domain11Code and data available at https://github.com/ostadabbas/Infant-Upper-Body-Postural-Symmetry..

Evaluating autonomous ground vehicles requires evaluating their mobility performance. Since autonomous vehicles are envisioned to make decisions in a variety of situations and environments too diverse to practically assess with only physical testing, their development, and evaluation will necessarily include the use of simulations. These simulations must represent reality sufficiently to represent the decisions that the vehicles would make in real-world. In this paper we present our Scoping Autonomous Vehicle Simulation (SAVeS) platform for benchmarking the performance of simulated environments for autonomous ground vehicle testing1.

Bilateral postural symmetry plays a key role as a potential risk marker for autism spectrum disorder (ASD) and as a symptom of congenital muscular torticollis (CMT) in infants, but current methods of assessing symmetry require laborious clinical expert assessments. In this paper, we develop a computer vision based infant symmetry assessment system, leveraging 3D human pose estimation for infants. Evaluation and calibration of our system against ground truth assessments is complicated by our findings from a survey of human ratings of angle and symmetry, that such ratings exhibit low inter-rater reliability. To rectify this, we develop a Bayesian estimator of the ground truth derived from a probabilistic graphical model of fallible human raters. We show that the 3D infant pose estimation model can achieve 68% area under the receiver operating characteristic curve performance in predicting the Bayesian aggregate labels, compared to only 61% from a 2D infant pose estimation model and 60% from a 3D adult pose estimation model, highlighting the importance of 3D poses and infant domain knowledge in assessing infant body symmetry. Our survey analysis also suggests that human ratings are susceptible to higher levels of bias and inconsistency, and hence our final 3D pose-based symmetry assessment system is calibrated but not directly supervised by Bayesian aggregate human ratings, yielding higher levels of consistency and lower levels of inter-limb assessment bias1.

Computer vision field has achieved great success in interpreting semantic meanings from images, yet its algorithms can be brittle for tasks with adverse vision conditions and the ones suffering from data/label pair limitation. Among these tasks is in-bed human pose monitoring with significant value in many healthcare applications. In-bed pose monitoring in natural settings involves pose estimation in complete darkness or full occlusion. The lack of publicly available in-bed pose datasets hinders the applicability of many successful human pose estimation algorithms for this task. In this paper, we introduce our Simultaneously-collected multimodal Lying Pose (SLP) dataset, which includes in-bed pose images from 109 participants captured using multiple imaging modalities including RGB, long wave infrared (LWIR), depth, and pressure map. We also present a physical hyper parameter tuning strategy for ground truth pose label generation under adverse vision conditions. The SLP design is compatible with the mainstream human pose datasets; therefore, the state-of-the-art 2D pose estimation models can be trained effectively with the SLP data with promising performance as high as 95% at [email protected] on a single modality. The pose estimation performance of these models can be further improved by including additional modalities through the proposed collaborative scheme.

We lay the groundwork for research in the algorithmic comprehension of infant faces, in anticipation of applications from healthcare to psychology, especially in the early prediction of developmental disorders. Specifically, we introduce the first-ever dataset of infant faces annotated with facial landmark coordinates and pose attributes, demonstrate the inadequacies of existing facial landmark estimation algorithms in the infant domain, and train new state-of-the-art models that significantly improve upon those algorithms using domain adaptation techniques. We touch on the closely related task of facial detection for infants, and also on a challenging case study of infrared baby monitor images gathered by our lab as part of in-field research into the aforementioned developmental issues1

We introduce deep switching auto-regressive factorization (DSARF), a deep generative model for spatio-temporal data with the capability to unravel recurring patterns in the data and perform robust short- and long-term predictions. Similar to other factor analysis methods, DSARF approximates high dimensional data by a product between time dependent weights and spatially dependent factors. These weights and factors are in turn represented in terms of lower dimensional latent variables that are inferred using stochastic variational inference. DSARF is different from the state-of-the-art techniques in that it parameterizes the weights in terms of a deep switching vector auto-regressive likelihood governed with a Markovian prior, which is able to capture the non-linear inter-dependencies among weights to characterize multimodal temporal dynamics. This results in a flexible hierarchical deep generative factor analysis model that can be extended to (i) provide a collection of potentially interpretable states abstracted from the process dynamics, and (ii) perform short- and long-term vector time series prediction in a complex multi-relational setting. Our extensive experiments, which include simulated data and real data from a wide range of applications such as climate change, weather forecasting, traffic, infectious disease spread and nonlinear physical systems attest the superior performance of DSARF in terms of long- and short-term prediction error, when compared with the state-of-the-art methods.

Infant motion analysis is a topic with critical importance in early childhood development studies. However, while the applications of human pose estimation have become more and more broad, models trained on large-scale adult pose datasets are barely successful in estimating infant poses due to the significant differences in their body ratio and the versatility of their poses. Moreover, the privacy and security considerations hinder the availability of adequate infant pose data required for training of a robust model from scratch. To address this problem, this paper presents (1) building and publicly releasing a hybrid synthetic and real infant pose (SyRIP) dataset with small yet diverse real infant images as well as generated synthetic infant poses and (2) a multi-stage invariant representation learning strategy that could transfer the knowledge from the adjacent domains of adult poses and synthetic infant images into our fine-tuned domain-adapted infant pose (FiDIP) estimation model. In our ablation study, with identical network structure, models trained on SyRIP dataset show noticeable improvement over the ones trained on the only other public infant pose datasets. Integrated with pose estimation backbone networks with varying complexity, FiDIP performs consistently better than the fine-tuned versions of those models. One of our best infant pose estimation performers on the state-of-the-art DarkPose model shows mean average precision (mAP) of 93.611The code is available at: github.com/ostadabbas/Infant-PoseEstimation. The SyRIP dataset can be downloaded at: Synthetic and Real Infant Pose (SyRIP)..

Factor analysis methods have been widely used in neuroimaging to transfer high dimensional imaging data into low dimensional, ideally interpretable representations. However, most of these methods overlook the highly nonlinear and complex temporal dynamics of neural processes when factorizing their imaging data. In this paper, we present deep Markov factor analysis (DMFA), a generative model that employs Markov property in a chain of low dimensional temporal embeddings together with spatial inductive assumptions, all related through neural networks, to capture temporal dynamics in functional magnetic resonance imaging (fMRI) data, and tackle their high spatial dimensionality, respectively. Augmented with a discrete latent, DMFA is able to cluster fMRI data in its low dimensional temporal embedding with regard to subject and cognitive state variability, therefore, enables validation of a variety of fMRI-driven neuroscientific hypotheses. Experimental results on both synthetic and real fMRI data demonstrate the capacity of DMFA in revealing interpretable clusters and capturing nonlinear temporal dependencies in these high dimensional imaging data.

Hospital-acquired pressure injuries (PI) are associated with longer hospital stays, pain, infection, and higher care costs. The traditional assessment techniques such as Braden scale, the most widely used PI risk assessment tool, lack predictive power. This study implements a machine learning algorithm using XGBoost and Braden subscales as its input features for PI risk assessment in intensive unit care (ICU) patients. We have evaluated our proposed PI risk assessment algorithm on a test dataset of 2,657 patients (PI prevalence equals to 17.57%) and have obtained 5.9% and 3.1% improvement in sensitivity and specificity respectively for our machine learning-based approach compared to the Braden scale.

All modern emotion theoretical views assume a role for peripheral physiological changes during emotional experiences. In this paper, we explored the correlation between autonomically-mediated changes in multimodal bodily signals and discrete emotional states. In order to fully exploit the information in each modality, week learners based on individual signal modalities are built and then fused to formed a robust inference model. To validate our model, three specific physiological signals including Electromyogram (EMG), Blood Volume Pressure (BVP) and Galvanic Skin Response (GSR) recorded during eight emotional states were analyzed. Our approach showed 88.1% emotion recognition accuracy, which outperformed the conventional Support Vector Machine (SVM) classifier with 17% accuracy improvement. Furthermore, in order to avoid information redundancy and the resultant over-fitting, a feature reduction method is proposed based on a correlation analysis to optimize the number of features required for training and validating each weak learner. Despite the feature space dimensionality reduction from 27 to 18 features, our methodology preserved the recognition accuracy of about 85.0%.

Automatic tracking of rodents' behaviors over time in their home cages is of great interest in psycho-physiological studies. The commercially-available animal monitoring systems use RGB videos or bio-potential signals to monitor behaviors of animals when exploring their surroundings. The based models of these devices starts from several thousands of dollars and the cost would increase if extra analysis features were added. In this study, we present a low-cost, non-contact animal tracking system which records depth data from the caged rodent to detect the animal's location and pose over time. An adaptive Gaussian Mixture Model (GMM) algorithm is employed to detect animal's center of mass and extract its movement trajectory over an extended period of time. The animal's pose is determined by applying Principle Component Analysis (PCA) on 3D depth data of the located animal. In conjunction with our previously-introduced respiratory detection algorithm, this system can be utilized as an automatic long-term and unobtrusive monitoring system for animal experiments. We validated the tracking accuracy of our system by monitoring two different caged voles. The voles' locations were correctly detected in 80% of times, while the poses were detected correctly in 100% of times confirmed by visually inspecting the color-coded depth videos.

The performance of portable and wearable biosensors is highly influenced by motion artifact. In this paper, a novel real-time adaptive algorithm is proposed for accurate motion-tolerant extraction of heart rate (HR) and pulse oximeter oxygen saturation ( SpO2) from wearable photoplethysmographic (PPG) biosensors. The proposed algorithm removes motion artifact due to various sources including tissue effect and venous blood changes during body movements and provides noise-free PPG waveforms for further feature extraction. A two-stage normalized least mean square adaptive noise canceler is designed and validated using a novel synthetic reference signal at each stage. Evaluation of the proposed algorithm is done by Bland-Altman agreement and correlation analyses against reference HR from commercial ECG and SpO2 sensors during standing, walking, and running at different conditions for a single- and multisubject scenarios. Experimental results indicate high agreement and high correlation (more than 0.98 for HR and 0.7 for SpO2 extraction) between measurements by reference sensors and our algorithm.

Pressure ulcer is a major problem for bed-bound and wheelchair-bound individuals specially in regions like sacrum, buttocks, hip, heels, back and head. Once developed, it is extremely uncomfortable and costly. Identification and monitoring of high-risk regions and their pressure distributions help nurses to have information about risk in each specific area of body and reposition patient efficiently. In this paper, we propose an algorithm to detect regions that are under high stress. Because of low resolution nature of pressure image and changes in shape of human body parts in various images, we adopted image processing algorithms. The image of human body is segmented using Delaunay triangulation. The extracted tree is compared to defined template for each posture. Then, signal processing and graph matching algorithms are used to label the tree according to the template. Pressure values of each specific region are collected for other phases of ulcer management such as risk assessment and reposition schedule. The experimental results indicate that our method can detect 9 (6) regions in supine (side) postures with average accuracy of 85.7%.

The foot complications constitute a tremendous challenge for diabetic patients, caregivers, and the healthcare system. With current technology, in-shoe monitoring systems can be implemented to continuously monitor foot's at-risk ulceration sites and send feedback to patients and physicians. The few available high resolution in-shoe pressure measuring systems are extremely expensive and targeting clinical use only. The more affordable price ranges can be reached by limiting the number of sensors in the shoe. Precise subject-specific sensor placement is still a challenge in such platforms. Moreover, there is no good way to estimate pressure on other points of the foot. In this paper, we address these technical challenges by proposing SCPM algorithm that reconstructs a continuous foot plantar pressure image from a sparse set of sensor readings. Using our technique, sensor placement can be the same in every electronic insole. However, the SCPM's trained parameters are unique for every subject and foot.

Pressure ulcer is a critical problem for bed-ridden and wheelchair-bound patients, diabetics, and the elderly. Patients need to be regularly repositioned to prevent excessive pressure on a single area of body, which can lead to ulcers. Pressure ulcers are extremely costly to treat and may lead to several other health problems, including death. The current standard for prevention is to reposition at-risk patients every two hours. Even if it is done properly, a fixed schedule is not sufficient to prevent all ulcers. Moreover, it may result in nurses being overworked by turning some patients too frequently. In this paper, we present an algorithm for finding a nurse-effort optimal repositioning schedule that prevents pressure ulcer formation for a finite planning horizon. Our proposed algorithm uses data from a commercial pressure mat assembled on the beds surface and provides a sequence of next positions and the time of repositioning for each patient.

Pressure ulcer is a severe threat for immobilized and peripheral neuropathic patients such as bed-ridden, elderly, and diabetics. Once developed, the complication of pressure ulcer causes pain, suffering, and longer hospitalization for the patients. Additionally, pressure ulcer management imposes a serious burden on the health care providers. The optimal strategy to deal with pressure ulcers is prevention. The current standard for prevention is to reposition at-risk patients every two hours. But, each patient has different needs based on overall vulnerability and damaged skin areas. A fixed schedule may either result in some patients getting ulcers, or nurses being overworked by turning some patients too frequently. In this paper, we present an efficient algorithm to find a repositioning schedule for bed- bound patients based on their risk of ulcer development. Our proposed algorithm uses data from a commercial pressure mat assembled on the bed's surface and provides a sequence of next positions and the time of repositioning for each patient. Our patient-specific turning schedule minimizes the overall cost of nursing staff involvement in repositioning the patients while simultaneously decreases the chance of pressure ulcer formation.

Monitoring human movements using wireless sensory devices promises to revolutionize the delivery of healthcare services. Such platforms use inertial information of their subjects for motion analysis. Potentially, each action or disease can be discovered by collaborative processing of sensor data from multiple locations on the body. This functionality is provided by a Body Sensor Network (BSN), which consists of several wireless sensor nodes positioned on different parts of the body. In spite of the revolutionary potential of this platform, power requirements and wearability have limited the commercialization of these systems. In this paper, we present an energy-efficient communication model for BSN applications which uses buffers to limit communication to short bursts, decreasing power usage and simplifying the communication. We formulate an optimization problem to reduce transmissions among sensor nodes and present an ILP-based solution and a fast greedy heuristic algorithm. We show that despite the decreased transmission efficiency, our greedy algorithm can be adopted for fast allocation of buffers in real-time. We experimentally compare the performance of both of the proposed approaches to the performance of an unbuffered system. Our results demonstrate that ILP and greedy solutions can reduce the amount of transmissions by an average factor of 70 and 41, respectively.