This paper presents a novel framework for Human Activity Recognition (HAR) by unifying all sensor streams, visual, audio, and inertial, into a single textual domain, enabling the direct application of GPT-3 for multimodal data classification. Unlike traditional pipelines that use dedicated encoders for each modality, we show that converting sensor outputs into text tokens offers both simplicity and a powerful proof-of-concept for large language models (LLMs). To further boost performance, we introduce a composite loss function combining cross-entropy, Kullback-Leibler divergence, total variation, and multimodal consistency terms, ensuring both temporal smoothness and cross-modal alignment. We conduct extensive experiments on the CMU-MMAC dataset, achieving up to 98% accuracy and significantly outperforming baseline methods. We also demonstrate robustness under missing sensor streams via partial tokenization, maintaining strong performance despite sensor failures. These results highlight the potential of LLM-driven HAR for enhanced human-robot interaction in real-world scenarios, and pave the way for broader multimodal applications of next-generation language models.

Human Activity Recognition (HAR) plays a crucial role in applications such as healthcare, smart environments, and human-robot interaction. This study proposes a novel Hyperbolic optimization strategy to improve model generalization by leveraging the geometric structure of the parameter space. To evaluate its effectiveness while also benefiting from the capabilities of modern sequence models in capturing long-range dependencies and multimodal interactions, the loss function is integrated into Transformer and GPT-2 models, fine-tuned on both unimodal (UCI-HAR, Opportunity) and multimodal (UTD-MHAD, NTU RGB+D) datasets. Unlike prior work that typically leverages only one to three modalities, this study utilizes all available modalities—RGB, depth, skeleton, and inertial—for multimodal evaluation. The Transformer achieves 98.26% and 93.40% accuracy on UCI-HAR and Opportunity, respectively, and 99.08% and 89.93% on UTD-MHAD and NTU RGB+D. GPT-2 also performs competitively, achieving 86.33% and 83.57% on the unimodal datasets, and 83.23% and 86.51% on the multimodal ones. These results highlight the potential of hyperbolic optimization for HAR across diverse sensor modalities and architectures.

This paper presents an innovative approach to enhancing Generative Adversarial Networks (GANs) for human face image generation. Generative tasks are traditionally hindered by issues like mode collapse and unstable training. We introduce a groundbreaking integration of Nash equilibrium principles into GAN architectures, featuring a tailored loss function that significantly stabilizes the training process. Our method not only bolsters GAN stability but also substantially improves the quality and variety of generated expressions. Rigorous experimentation across benchmark datasets like CIFAR-10, CelebA, and ImageNet confirms the exceptional performance of our model, particularly evidenced by significant reductions in the Fréchet Inception Distance, a testament to the method’s efficacy in producing more realistic and diverse facial expressions. The implications of this research are far-reaching, particularly in the realm of human-machine interaction where expressive and diverse facial expressions are crucial. We will extend the application of the proposed method to other domains beyond facial expression generation, such as medical imaging or autonomous vehicle perception systems, to assess its adaptability and effectiveness.

Human activity anticipation is a pillar of new interactive multimedia technologies. It enables better human-computer interaction and creative sensory experience in all types of virtual reality-based multimedia applications. Human behaviors are hierarchically constituted in terms of low-level and high-level activities. Fine-grained activities play a substantial role in the early recognition of simple and complex human movements in real-time with low latency, allowing longer predictions in the future. In this paper, a new objective function is proposed to anticipate fine-grained human activities using IMU data, which still suffers from an imbalance problem. Four customized objective surrogate functions are applied in unidirectional and bidirectional recurrent neural networks and compared to efficiently optimize the model considering the loss of individual classes. The experiments on five datasets across proposed loss functions show that the proposed model significantly outperforms the counterpart methods and advances the state-of-the-art. The proposed model shows accuracy scores of up to 98% and 96% for high-level and low-level activities, respectively.Note to Practitioners—There are a large number of methods that can be used for the recognition of human activity in assisted living, smart home technologies, and pervasive healthcare applications. However, studies on these methods tend to focus on recognition rather than anticipation of human activity. Furthermore, they use either vision-based or egocentric video data, which can make it challenging to collect and evaluate in virtual reality-based scenarios. To enable more natural interaction with physical and virtual reality environments, virtual reality-based technologies require built-in algorithms capable of consistently identifying complex and varied human movements. Fine-grained activities are paramount in predicting activities at an early stage to adequately characterize human behaviors. In this paper, a study is conducted to anticipate fine-grained human activity based on IMU (Inertial Measurement Unit) data, which are still subject to the negative effects of an imbalanced sample distribution. The proposed model is evaluated on different benchmarks and compared with baseline methods. The results show that the proposed model outperforms all baselines. These results have important implications for human activity prediction problems.

Post-traumatic stress disorder (PTSD) is a mental disorder that can be developed after witnessing or experiencing extremely traumatic events. PTSD can affect anyone, regardless of ethnicity, or culture. An estimated one in every eleven people will experience PTSD during their lifetime. The Clinician-Administered PTSD Scale (CAPS) and the PTSD Check List for Civilians (PCL-C) interviews are gold standards in the diagnosis of PTSD. These questionnaires can be fooled by the subject's responses. This work proposes a deep learning-based approach that achieves state-of-the-art performances for PTSD detection using audio recordings during clinical interviews. Our approach is based on MFCC low-level features extracted from audio recordings of clinical interviews, followed by deep high-level learning using a Stochastic Transformer. Our proposed approach achieves state-of-the-art performances with an RMSE of 2.92 on the eDAIC dataset thanks to the stochastic depth, stochastic deep learning layers, and stochastic activation function.

In recent years, semantic segmentation has become an increasingly important task in computer vision, with numerous applications in image analysis and object recognition. One popular approach for semantic segmentation is the use of convolutional neural networks (CNNs), such as the U-net architecture, which is known for its ability to capture both local and global contexts in images. However, the U-Net model can be computationally intensive and time-consuming, especially for large or high-resolution images. This paper proposes a depth-residual separable U-net (DRSU-net) model to improve the U-net model for semantic segmentation tasks. The proposed approach introduces an additional regularisation technique instead of dropout and L2 regularisation and modifies the network's structure to reduce the risk of overfitting and the number of model parameters. This reduces training time without losing model information, especially for large datasets. The effectiveness of the DRSU-net method is demonstrated through extensive experiments on several semantic segmentation benchmarks. The results show improved performance and lower temporal complexity compared to the basic U-net and state-of-the-art architectures.

In this work, the results of an exploratory study into the use of the publicly available GPT-2 model for simultaneous conditional human activity signal synthesis and classification are presented. To accomplish this, the small variant of a pre-trained GPT-2 model is fine-tuned on quantized and windowed human activity signal sequences. The conditional generation is achieved by appending and prepending class labels to each window to introduce class information. During the generation phase, the model is prompted either with a class label for synthesizing signals, or a signal sequence for classification by synthesizing a class label. The study is conducted on the publicly available WISDM and UCI-HAR datasets, and the generated signals are evaluated using an LSTM-CNN model. As a classifier, the fine-tuned GPT-2 model achieved an overall accuracy of 90.47% on the WISDM, and 82.29% on the UCI-HAR dataset, which is 4% and 6% lower than the LSTM-CNN model evaluated on the same test subsets. When used as a multi-class generator, on average, 86.33% of the generated data are classified as valid samples by the LSTM-CNN model. While there is room for improvement, these results demonstrate that GPT family architectures can be used for simultaneous conditional signal synthesis and classification, opening up new opportunities for human activity recognition.

The advent of the internet of things (IoT) and artificial intelligence (AI) technologies enable continuous data generation, also known as data streams. Considering data streams is required for ambient intelligence (AmI) systems for context-aware assistance services. This paper proposes a hybrid approach combining deep learning models and probabilistic commonsense reasoning over data streams to anticipate human activities. The reasoning is performed using the event calculus formulated in answer set programming (ECASP); the latter allows for abductive and temporal reasoning, which enables an eXplainable AI (XAI) approach. An activity context ontology is exploited for the reasoning axiomatisation. Several experiments were conducted to demonstrate the effectiveness of the proposed approach in terms of accuracy and computation time.

Gesture recognition becomes a thriving research area in modern human motion recognition systems. The intensification of demands on efficient interactive human-machine-interface systems, commercial objectives, and many other factors contributes to fuel this revival dynamics. Understanding human gestures becomes essential for prevention and health monitoring applications. In particular, analyzing hand gestures is of paramount importance in personalized healthcare-related applications to help practitioners providing more qualitative assessments of subject's pathologies, such as Parkinson's diseases. This work proposes a novel deep neural network approach to forecast future gestures from a given sequence of hand motion using a wearable capacitance sensor of an innovative gesture recognition hardware system. To do this, we use an attention-based recurrent neural network to capture the temporal features of hand motion to unveil the underlying pattern between the gesture and these sequences. While the attention layers capture patterns from the weights of the short term, the gated recurrent unit (GRU) neural network layer learns the inherent interdependency of long-term hand gesture temporal sequences. The efficiency of the proposed model is evaluated with respect to cutting-edge work in the field using several metrics. Note to Practitioners-In this article, the problem of human hand gesture recognition is analyzed using deep learning techniques. The proposed model uses input historical motion sequences collected from a wearable capacitance sensor to predict hand gestures. The model leverages the intrinsic correlation of motion sequences and extracts the salient part of the sequences by taking into consideration their temporal, complex, and nonlinear features. The approach studies the effect of different lengths of historical motion sequences in prediction outcomes. This allows for avoiding using cumbersome data collection, heavy data treatment, and high computational cost. The model performance is trained and assessed on real-world data by performing comparisons with alternative approaches, including well-known classifiers. The model yields very encouraging results and demonstrates that the proposed approach is quite competitive as it can reproduce typical activity trends for important channels. The present findings could help in the development of intelligent wearable devices for predicting hand gestures using a limited number of channels. This work could also help practitioners to provide a more qualitative appraisal of patients suffering from different pathologies such as Parkinson's diseases to personalized healthcare-related applications and to develop wearable gesture recognition devices on a large scale.

Communication-enabled devices routinely carried by individuals have become pervasive, opening unprecedented opportunities for collecting digital metadata about the mobility of large populations. In this paper, we propose a novel methodology for the estimation of people density at metropolitan scales, using subscriber presence metadata collected by a mobile operator. Our approach suits the estimation of static population densities, i.e., of the distribution of dwelling units per urban area contained in traditional censuses. More importantly, it enables the estimation of dynamic population densities, i.e., the time-varying distributions of people in a conurbation. By leveraging substantial real-world mobile network metadata and ground-truth information, we demonstrate that the accuracy of our solution is superior to that granted by state-of-the-art methods in practical heterogeneous urban scenarios.

Smartphones and other mobile devices are today pervasive across the globe. As an interesting side effect of the surge in mobile communications, mobile network operators can now easily collect a wealth of high-resolution data on the habits of large user populations. The information extracted from mobile network traffic data is very relevant in the context of population mapping: it provides a tool for the automatic and live estimation of population densities, overcoming the limitations of traditional data sources such as censuses and surveys. In this paper, we propose a new approach to infer population densities at urban scales, based on aggregated mobile network traffic metadata. Our approach allows estimating both static and dynamic populations, achieves a significant improvement in terms of accuracy with respect to state-of-the-art solutions in the literature, and is validated on different city scenarios.

Research on guidance and method engineering has highlighted that many method engineering issues, such as lack of flexibility or adaptation, are solved more effectively when intentions are explicitly specified. However, software engineering process models are most often described in terms of sequences of activities. This paper presents a novel approach, so-called Map Miner Method (MMM), designed to automate the construction of intentional process models from process logs. To do so, MMM uses Hidden Markov Models to model users' activities logs in terms of users' strategies. MMM also infers users' intentions and constructs fine-grained and coarse-grained intentional process models with respect to the Map metamodel syntax (i.e., metamodel that specifies intentions and strategies of process actors). These models are obtained by optimizing a new precision-fitness metric. The result is a software engineering method process specification aligned with state of the art of method engineering approaches. As a case study, the MMM is used to mine the intentional process associated to the Eclipse platform usage. Observations show that the obtained intentional process model offers a new understanding of software processes, and could readily be used for recommender systems.

So far, process mining techniques have suggested to model processes in terms of tasks that occur during the enactment of a process. However, research on method engineering and guidance has illustrated that many issues, such as lack of flexibility or adaptation, are solved more effectively when intentions are explicitly specified. This paper presents a novel approach of process mining, called Map Miner Method (MMM). This method is designed to automate the construction of intentional process models from process logs. MMM uses Hidden Markov Models to model the relationship between users' activities logs and the strategies to fulfill their intentions. The method also includes two specific algorithms developed to infer users' intentions and construct intentional process model (Map) respectively. MMM can construct Map process models with different levels of abstraction (fine-grained and coarse-grained process models) with respect to the Map metamodel formalism (i.e., metamodel that specifies intentions and strategies of process actors). This paper presents all steps toward the construction of Map process models topology. The entire method is applied on a large-scale case study (Eclipse UDC) to mine the associated intentional process. The likelihood of the obtained process model shows a satisfying efficiency for the proposed method.

Abstract: The paper addresses the problem of the low usage of software development methods in software development practice. This has been recognized as one of the key reasons for failures in software development projects and a contributor to the low quality of software. We introduce a novel approach that could help to improve the maturity of software development processes. The approach is based on the method engineering principles taking into account the limitations that hinder its use in practice. The main objective of our research is to show that the method engineering concepts are applicable in real settings and that could contribute to the higher quality of software development processes and their products. 1

Nowadays, digital traces are omnipresent in Information System (IS). Companies track IS interactions to retrieve and compile information about actors. Researchers of various streams, within IT and beyond, focused on recording actor interactions with systems and the technical possibilities to identify record and store these interactions. Tracing functionality has appeared in almost all common computer applications. This PhD project will focus on the establishment of a trace-based system and propose recommendations to actors regarding to their context. The objective of this thesis is to study process traces to propose recommendations to the actors by identifying a set of generic processes adaptable to the current actors' context. Thus, any actor, expert or novice, will be able to use this knowledge that gives contextual clues to identify the potential steps he could perform.

Since several decades, discovering process models is a subject of interest in the Information System (IS) community. Approaches have been proposed to recover process models, based on the recorded sequential tasks (traces) done by IS's actors. However, these approaches only focused on activities and the process models identified are, in consequence, activity-oriented. Intentional process models focus on the intentions underlying activities rather than activities, in order to offer a better guidance through the processes. Unfortunately, the existing process-mining approaches do not take into account the hidden aspect of the intentions behind the recorded user activities. We think that we can discover the intentional process models underlying user activities by using Intention mining techniques. The aim of this paper is to propose the use of probabilistic models to evaluate the most likely intentions behind traces of activities, namely Hidden Markov Models (HMMs). We focus on this paper on a supervised approach that allows discovering the intentions behind the user activities traces and to compare them to the prescribed intentional process model.