We present V-HPOT, a novel approach for improving the cross-domain performance of 3D hand pose estimation from egocentric images across diverse, unseen domains. State-of-the-art methods demonstrate strong performance when trained and tested within the same domain. However, they struggle to generalise to new environments due to limited training data and depth perception – overfitting to specific camera intrinsics. Our method addresses this by estimating keypoint z-coordinates in a virtual camera space, normalised by focal length and image size, enabling camera-agnostic depth prediction. We further leverage this invariance to camera intrinsics to propose a self-supervised test-time optimisation strategy that refines the model’s depth perception during inference. This is achieved by applying a 3D consistency loss between predicted and in-space scale-transformed hand poses, allowing the model to adapt to target domain characteristics without requiring ground truth annotations. V-HPOT significantly improves 3D hand pose estimation performance in cross-domain scenarios, achieving a 71% reduction in mean pose error on the H2O dataset and a 41% reduction on the AssemblyHands dataset. Compared to state-of-the-art methods, V-HPOT outperforms all single-stage approaches across all datasets and competes closely with two-stage methods, despite needing ≈ ×3.5 to ×14 less data. https://github.com/wiktormucha/vhpot.

WiFi-based human activity recognition (HAR) faces significant challenges in cross-domain generalization due to dynamic environmental variations, device heterogeneity, and subtle changes in human behavior. In this paper, we introduce DATTA – Domain-Adversarial Test-Time Adaptation – a novel framework that combines domain-adversarial training (DAT) with test-time adaptation (TTA) and a random weight-resetting mechanism. Unlike previous approaches that apply these techniques in isolation, DATTA is specifically tailored for WiFi-based HAR: it leverages DAT to learn robust, domain-invariant features while TTA continuously refines the model on streaming data. To mitigate catastrophic forgetting during adaptation, we incorporate a weight-resetting mechanism, ensuring sustained performance over prolonged domain shifts. Our extensive experiments on the Widar3.0-G6D dataset demonstrate that DATTA not only outperforms state-of-the-art methods by up to 8.1% in F1-Score but also achieves real-time inference with a lightweight architecture, making it a compelling solution for practical WiFi sensing applications. The PyTorch implementation of DATTA is publicly available at: https://github.com/StrohmayerJ/DATTA.

The identification of ethical impacts is the first phase of Ethical Impact Assessments, which are used to evaluate the ethical implications of new technologies. Using a structured methodology, this paper explores the identification of ethical impacts of a video-based tool for monitoring dementia-related behaviours. By reflecting on autonomy, dignity, non-maleficence, beneficence, justice and privacy, our work contributes to a broader understanding of the ethical landscape surrounding the use of Artificial Intelligence (AI) and computer vision in healthcare. Our contribution goes beyond the specific aspects of this system as we address consent, privacy and autonomy, factors that are relevant to any technology for people with dementia. We call for the integration of ethical considerations into the design and implementation of AI technologies, ensuring that innovations not only advance clinical care, but also respect fundamental ethical principles.

Action recognition is essential for egocentric video understanding, allowing automatic and continuous monitoring of Activities of Daily Living (ADLs) without user effort. Existing literature focuses on 3D hand pose input, which requires computationally intensive depth estimation networks or wearing an uncomfortable depth sensor. In contrast, there has been insufficient research in understanding 2D hand pose for egocentric action recognition, despite the availability of user-friendly smart glasses in the market capable of capturing a single RGB image. Our study aims to fill this research gap by exploring the field of 2D hand pose estimation for egocentric action recognition, making two contributions. Firstly, we introduce two novel approaches for 2D hand pose estimation, namely EffHandNet for single-hand estimation and EffHandEgoNet, tailored for an egocentric perspective, capturing interactions between hands and objects. Both methods outperform state-of-the-art models on H2O and FPHA public benchmarks. Secondly, we present a robust action recognition architecture from 2D hand and object poses. This method incorporates EffHandEgoNet, and a transformer-based action recognition method. Evaluated on H2O and FPHA datasets, our architecture has a faster inference time and achieves an accuracy of 91.32% and 94.43%, respectively, surpassing state of the art, including 3D-based methods. Our work demonstrates that using 2D skeletal data is a robust approach for egocentric action understanding. Extensive evaluation and ablation studies show the impact of the hand pose estimation approach, and how each input affects the overall performance. The code is available at https://github.com/wiktormucha/effhandegonet.

WiFi Channel State Information (CSI)-based Human Activity Recognition (HAR) enables contactless, long-range sensing in spatially constrained environments while preserving visual privacy. However, despite the ubiquity of WiFi-enabled devices, few expose CSI, limiting sensing hardware options. Variants of the Espressif ESP32 have emerged as potential compact, low-cost, and easy-to-deploy solutions for WiFi CSI-based HAR. In this work, four ESP32-S3-based 2.4 GHz directional antenna systems are evaluated for their ability to facilitate long-range through-wall HAR. Two promising systems are identified: one combines ESP32-S3 with a directional biquad antenna, and the second uses the built-in printed inverted-F antenna (PIFA) achieving directionality through a plane reflector. In a comprehensive evaluation of line-of-sight (LOS) and non-line-of-sight (NLOS) HAR performance, both systems are deployed in an office environment spanning a distance of 18 meters across five rooms. In this experimental setup, the Wallhack 1.8 k dataset, comprising 1,806 CSI amplitude spectrograms of human activities, is collected and made publicly available. Based on Wallhack1.8k, activity recogn tion models using the EfficientNetV2 architecture are trained to assess system performance in LOS and NLOS scenarios. For the core NLOS activity recognition problem, the biquad antenna and PIFA-based systems achieve accuracies of 92.0 ± 3.5 and 86.8 ± 4.7, respectively, demonstrating the feasibility of long-range through-wall HAR.

This work presents a seminal approach for synthesizing images from WiFi Channel State Information (CSI) in through-wall scenarios. Leveraging the strengths of WiFi, such as cost-effectiveness, illumination invariance, and wall-penetrating capabilities, our approach enables visual monitoring of indoor environments beyond room boundaries and without the need for cameras. More generally, it improves the interpretability of WiFi CSI by unlocking the option to perform image-based downstream tasks, e.g., visual activity recognition. In order to achieve this crossmodal translation from WiFi CSI to images, we rely on a multimodal Variational Autoencoder (VAE) adapted to our problem specifics. We extensively evaluate our proposed methodology through an ablation study on architecture configuration and a quantitative/qualitative assessment of reconstructed images. Our results demonstrate the viability of our method and highlight its potential for practical applications.

We present FUS3D, a fast and lightweight system for real-time 3D object detection and tracking on edge devices. Our approach seamlessly integrates stages for 3D object detection and multi-object-tracking into a single, end-to-end trainable model. FUS3D is specially tuned for indoor 3D human behavior analysis, with target applications in Ambient Assisted Living (AAL) or surveillance. The system is optimized for inference on the edge, thus enabling sensor-near processing of potentially sensitive data. In addition, our system relies exclusively on the less privacy-intrusive 3D depth imaging modality, thus further highlighting the potential of our method for application in sensitive areas. FUS3D achieves best results when utilized in a joint detection and tracking configuration. Nevertheless, the proposed detection stage can function as a fast standalone object detection model if required. We have evaluated FUS3D extensively on the MIPT dataset and demonstrated its superior performance over comparable existing state-of-the-art methods in terms of 3D object detection, multi-object tracking, and, most importantly, runtime.

The recognition of supermarket products on mobile devices is gaining importance as more and more consumers seek to make informed decisions about their purchases in real time. However, the realization is often difficult due to the vast product assortments of modern supermarkets and the limited computational resources available on mobile devices. In this work, we propose a real-time on-device product recognition pipeline, based on the Global Trade Item Number (GTIN) system, that is both robust to dynamic changes in the product assortment and scalable to tens of thousands of products. We evaluate detection performance on SKU110k and R6k datasets and demonstrate the scalability of our pipeline with 5974 different products, using synthetic data. Furthermore, the proposed product recognition pipeline is deployed on a Google Pixel 6 mobile phone, where it achieves an inference time of 121ms (8.3fps), demonstrating its real-time capabilities in practice.

Developing automated tools for sustainable film preservation of extensive historical film collections assumes an understanding of fundamental cinematographic settings. In order to be able to investigate new approaches to detect and classify cinematographic settings, this paper proposes a novel large-scale historical film dataset with cinematographic annotations (HISTORIAN), i.e., shot boundaries, shot types, camera movements. The dataset consists of 98 digitized original analog film reels related to the Second World War and 10593 film shots manually annotated with human film experts. Moreover, annotations for overscan areas such as sprocket holes are included. A baseline film analysis pipeline is introduced and evaluated. To the best of our knowledge, HISTORIAN is the first dataset that covers the challenges and characteristics of historical film documentaries and provides novel possibilities for exploring automatic film analysis tools.

Depth and thermal sensors are well-suited for computer vision applications that involve the continuous monitoring of people, particularly in combination. Yet there is limited research in this field and a lack of available datasets. We present a method for creating synthetic but realistic depth and thermal images that include sensor noise. We utilize this method to create the SDT dataset, which contains 40k image pairs including labels for person detection and pose classification, and is publicly available. To assess the quality our image synthesis method and the utility of SDT, we train CNNs for classification on the SDT dataset and evaluate them on real data. The CNNs achieve accuracies up to 98%, highlighting their ability to generalize from synthetic to real data.

We present a public dataset for Identity Preserved Tracking (IPT) consisting of sequences of depth data recorded using an Orbbec Astra depth sensor. The dataset features sequences in ten different locations with a high amount of background variation and is designed to be applicable to a wide range of tasks. Its labeling is versatile, allowing for tracking in either 3d space or image coordinates. Next to frame-by-frame 3d and inferred bounding box labeling we provide supplementary annotation of camera poses and room layouts, split in multiple semantically distinct categories. Intended use-cases are applications where both a high level understanding of scene understanding and privacy are central points of consideration, such as active and assisted living (AAL), security and industrial safety. Compared to similar public datasets IPT distinguishes itself with its sequential data format, 3d instance labeling and room layout annotation. We present baseline object detection results in image coordinates using a YOLOv3 network architecture and implement a background model suitable for online tracking applications to increase detection accuracy. Additionally we propose a novel volumetric non-maximum suppression (V-NMS) approach, taking advantage of known room geometry. Last we provide baseline person tracking results utilizing Multiple Object Tracking Challenge (MOTChallenge) evaluation metrics of the CVPR19 benchmark.

The purpose of this paper is the development of a method to extract gait parameters from an existing person tracker based on depth data. The underlying person tracking method integrates an algorithm for auto-calibration of the depth sensor, which allows plug-and-play installation. Based on the provided tracking information, gait velocity, distance, and duration is measured, as well as gait cycle components. The estimation of these components is based on a gait event demarcation approach using a wearable accelerometer, that provides similar signals as the person tracker. These signals are predicted by a random forest and post-processed using an optimization algorithm and a-priori information. To evaluate the system, we recorded a dataset with 10 healthy adults performing walks on 4 different paths. The detected gait cycle parts show deviations to the actual annotated components of 0.06-0.1s, and 0.69-9.42%. The velocity measurement achieves a mean error of 3.64cm/s, while the distance error results in 20.48cm, duration measurement obtains an average error of 0.27s.

Acute Myeloid Leukaemia (AML) is a rare type of childhood acute leukaemia. During treatment, the assessment of the number of cancer cells is particularly important to determine treatment response and consequently adapt the treatment scheme if necessary. Minimal Residual Disease (MRD) is a diagnostic measure based on Flow CytoMetry (FCM) data that captures the amount of blasts in a blood sample and is a clinical tool for planning patients' individual therapy, which requires reliable blast identification. In this work we propose a novel semi-supervised learning approach, which is acquired whenever large amounts of unlabeled data and only a small amount of annotated data is available. The proposed semi-supervised learning approach is based on Wasserstein Generative Adversarial Network (WGAN) latent space embeddings learned in an unsupervised fashion and a simple Fully connected Neural Network (FNN) trained on labeled data leveraging the learned embedding. We apply our proposed learning approach for semi-supervised classification of blasts vs. non-blasts. We compare our approach with two baseline approaches, 1) semi-supervised learning based on Principal Component Analysis (PCA) embedding, and 2) a deep FNN that is trained only on the annotated data without leveraging an embedding. Results suggest that our proposed semi-supervised WGAN embedding outperforms semi-supervised learning based on PCA embeddings and if only small amounts of annotated data is available it even outperforms an FNN classifier.

We present a method for jointly classifying objects in depth maps and regressing amodal (extending beyond occluded parts) 3D bounding boxes in a way that is highly robust to occlusions. Our method is based on a novel multi-view convolutional neural network architecture with shared layers for both tasks, improving efficiency. The network processes views that encode object geometry and occlusion information and outputs class scores and bounding box coordinates in world coordinates, requiring no post-processing steps. We demonstrate the effectiveness of our method by example of fall detection, presenting a new dataset of 40k samples rendered from 3D models. On this dataset, our method achieves an average classification accuracy above 97% and a regression error below 10 cm at occlusion ratios of up to 90%. The dataset and trained models are publicly available.

The problem of incorporating spatial information to the bag-of-visual-words model for image classification is addressed in this letter. To incorporate such information, we propose to encode the global geometric relationships of the visual words in the 2D image plane in a scale- and rotation-invariant manner. This is established by measuring scale- and rotation-invariant geometrical properties given by triangles of identical visual words. Experimental results demonstrate that our proposed method is more robust to changes in scale and image rotations than the bag-of-visual words model on a butterfly and fish dataset.

We propose to perform image-based ancient coin classification by recognizing symbols minted on the reverse side of coins. Dense sampling based bag-of-visual-words model is used for symbol recognition. The lack of spatial information in the bag-of-visual-words model degrades symbol recognition rate as the symbols have specific geometric structures. Furthermore, coins can be imaged under various rotations resulting in severely rotated symbols. Therefore we propose a novel bag-of-visual-wordsmodel for symbol-based coin classification which accounts for the spatial arrangement of the visual words in a rotation invariant manner. We perform our experiments on images collected from three different sources thus making our dataset more challenging. To evaluate our proposed model for robustness to rotations, we synthetically generated severely rotated coin images. In the presence of rotation differences between coins, our model outperforms the conventional bag-of-visual-words model as well as recently proposed angles histograms of pair-wise identical visual words model.

Unusual inactivity is caused by events, where elderly need help (e.g., falls, illness). In order to detect unusual behavior, modeling of activity results in inactivity profiles. State-of-the-Art approaches focus on temporal aspects of inactivity by only considering deviations of inactivity over time. This work proposes the use of spatial information in combination with temporal aspects to enhance the robustness and reduce the number of false alarms. The proposed approach is evaluated on two different datasets containing 100 days resp. 50 days of activity data of elderly people and results are compared to the State-of-the-Art.

Classification of ancient coins is a substantial part of numismatic research which needs a large amount of expert knowledge due to the high number of classes to be considered. In this paper we propose an automatic image-based classification method for ancient coins to support this time-consuming and difficult process. We demonstrate that previously proposed learning-based methods suffer from the practical conditions of this problem: a high number of classes, limited number of training samples per class and complex intra-class variations. As a solution we propose a similarity metric based on feature correspondence which is designed to be robust against the possible intra-class coin variations like degraded parts, non-rigid deformations and illumination-induced appearance changes. The similarity metric is used in an exemplar-based ancient coin classification scheme which shows to outperform previously proposed methods for ancient coin recognition. Experiments are conducted on a dataset of 60 Roman Republican coin classes where the presented method achieves classification rates ranging from 72.7% for the case of one training sample per class up to 97.2% when nine training samples per class are used.

This paper presents a novel method for dense 3D reconstruction of man-made environments. Such environments suffer from texture less and non-Lambertian surfaces, where conventional, feature-Based 3D reconstruction pipelines fail to obtain good feature matches. To compensate this lack of feature matches, we exploit the semantic information available in 2D images to estimate both a corresponding 3D position and a 3D surface normal for each pixel. A semantic classifier is therefore applied on a single segmented image in order to get a likelihood for a segment providing one of the surface normals within a discrete set of them. To improve the accuracy of this labeling step, we exploit multiple segmentation methods. The global best surface normal configuration over all pixels of an image is then obtained by using a Markov Random Field. In the last step, the 3D model of a single 2D input image is reconstructed by combining the semantic surface normal estimation with the sparse point cloud coming from feature Based matching. It is shown experimentally, that our proposed method clearly outperforms state-of-the-art dense 3D reconstruction pipelines and surface layout estimation approaches.

This paper presents a computational effective framework for tracking and pose estimation of vehicles in videos reaching comparable performance to state-of-the-art methods. We cast the problem of vehicle tracking as ranking possible poses for each frame and connecting subsequent poses by exploiting a feasible motion model over time. As a novelty, we use random forests trained on a set of existing 3D models for estimating the pose. We discretize the viewpoint space for training, where a synthetic camera is orbiting around the models. To compare projections of 3D models to real world 2D input frames, we introduce simple but discriminative principle gradient features to describe both images. A Markov Random Field ensures to pick the perfect pose over time and the vehicle to follow a feasible motion. As can be seen from our experiments performed on a variety of videos with vast variation of vehicle types, the proposed framework achieves similar results in less computational time compared to state-of-the-art methods.

In this paper, we propose a new approach for people detection using a relational feature model (RFM) in combination with histogram similarity functions such as the bhattacharyya distance, histogram intersection, histogram correlation and the chi-square χ2histogram similarity function. The relational features are computed for all combinations of extracted features from a feature detection algorithm such as the Histograms of Oriented Gradients (HOG) feature descriptor. Our experiments show, that the information of spatial histogram similarities reduces the number of false positives while preserving true positive detections. The detection algorithm is done, using a multi-scale overlapping sliding window approach. In our experiments, we show results for different sizes of the cell size from the HOG descriptor due to the large size of the resulting relational feature vector as well as different results from the mentioned histogram similarity functions. Additionally our results show, that in addition to less false positives, true positive responses in regions near people are much more accurate using the relational features compared to non-relational feature models.

In this paper we present a novel object classification and pose recovery algorithm which takes advantage of existing 3D models and multiple synchronized and calibrated views. Having a calibrated scenario provides redundant data which can be exploited for gathering spatial consistency of an object's 3D pose and its class. In a first step, the cameras need to be calibrated and aligned to one common coordinate system. A training set of 3D models, a calibrated setup and Harris corner features are used to find the best fitting 2D projection for an object within the scene. The results are improved by aligning multiple synchronized views to gain spatial consistency. Our experiments using real data show the enhanced results using a calibrated setup over analyzing each camera separately.

This paper deals with a novel method for object tracking. In the first step interest points are detected and feature descriptors around them are calculated. Sets of known points are created, allowing tracking based on point matching. The set representation is updated online at every tracking step. Our method uses one-shot learning with the first frame, so no offline and no supervised learning is required. Following an object recognition based approach there is no need for a background model or motion model, allowing tracking of abrupt motion and with non-stationary cameras. We compare our method to Mean Shift and Tracking via Online Boosting, showing the benefits of our approach.

This paper presents a novel approach for unexpected behavior recognition in image sequences with attention to high density crowd scenes. Due to occlusions, object-tracking in such scenes is challenging and in cases of low resolution or poor image quality it is not robust enough to efficiently detect abnormal behavior. The wide variety of possible actions performed by humans and the problem of occlusions makes action recognition unsuitable for behavior recognition in high density crowd scenes. The novel approach, which is presented in this paper uses features based on motion information instead of detecting actions or events in order to detect abnormality. Experiments demonstrate the potentials of the approach.

By analyzing the low level features of images only, skin detection in visual data cannot be solved. To compensate for this major drawback of many approaches, we combine a state of the art recognition algorithm with color model based skin detection. Detected faces in videos are the basis for adaptive skin-color models, which are propagated throughout the video, providing a more precise and accurate model in its recognition performance than pure color based approaches. The approach is able to run in real-time and does not need prior data-specific training. We received challenging online videos from an online service provider and use additional videos from public Web platforms covering a grand variety of different skin-colors, illumination circumstances, image quality and difficulty levels. In an extensive evaluation we estimated the best performing parameters and decide on the best model propagation techniques. We show that adaptive model propagation outperforms static low level detection.

Archaeology is at a point where it can benefit greatly from the application of computer vision methods, and in turn provides a large number of new, challenging and interesting conceptual problems and data for computer science. This is true in particular in the study of ceramics - the most abundant and widespread of all archaeological finds. The traditional way of documenting archaeological sherds is to draw the profile line, which is the intersection of a sherd along the axis of symmetry. A profilograph is a mechanical device, which can directly acquire and transfer a profile line by pin-pointing the profile on a sherd to a computer. We developed a fully automated vision system, which is able to compute the profile line out of the acquired 3D model of the fragment. In this paper we want to give a thorough comparison between the traditional manual approach, the profilograph and our system and present an improvement of the robustness of our approach by finding circular rills on the fragments. Practical experiments have been undertaken at the excavation Tel Dor in Israel.

Abstract Motivated by the current requirements of archaeologists, we are developing an automated archival system for archaeological classification and reconstruction of ceramics. Our system uses the profile of an archaeological fragment, which is the cross‐section of the fragment in the direction of the rotational axis of symmetry, to classify and reconstruct it virtually. Ceramic fragments are recorded automatically by a 3D measurement system based on structured (coded) light. The input data for the estimation of the profile is a set of points produced by the acquisition system. By registering the front and the back views of the fragment the profile is computed and measurements like diameter, area percentage of the complete vessel, height and width are derived automatically. We demonstrate the method and give results on synthetic and real data. Copyright © 2003 John Wiley & Sons, Ltd.

We are developing an automated classification and reconstruction system for archaeological fragments. The goal is to relate different fragments belonging to the same vessel based on shape, material and color, thus the color information is important in the pre-classification process. In this work a color specification technique is proposed, which exploits the fact that the spectral reflectance of materials like archaeological fragments vary slowly. We explain how the acquisition system is calibrated in order to get accurate colorimetric information with respect to archaeological requirements. Experimental results are presented for archaeological objects and for a set of test color patches.