This paper introduces Vistar, a novel self-supervised framework for inertial measurement unit (IMU) signal perception designed for large language models (LLMs). Unlike visual data, IMU signals are high-frequency time series with low interpretability, making manual annotation with natural language particularly challenging. Even when using vision-language models (VLMs) to describe events in videos synchronized with IMU signals, a semantic gap remains between high-level visual semantics and low-level IMU vibrations. The core idea of Vistar is to achieve accurate IMU signal perception through collaborations between offline cross-modal alignment and online retrieval-augmented generation. During offline training, Vistar uses pretrained vision and language encoders as anchors to learn IMU encoders via hierarchical cross-modal contrastive learning, establishing both inter- and intra-sample alignment. Given that the enhanced training strategy still fails to achieve precise alignment between IMU and text, during online inference, Vistar further employs a retrieval-augmented generation mechanism to generate distilled textual descriptions from similar text filtered based on structural relations of their paired IMU samples. Extensive evaluations on three multimodal datasets demonstrate that Vistar consistently outperforms state-of-the-art (SOTA) baselines by up to 57.45% in IMU-to-text retrieval and improves the generated text similarity with ground truths in IMU perception by up to 31.90%.

Standard multimodal self-supervised learning (SSL) algorithms regard cross-modal synchronization as implicit supervisory labels during pretraining, thus posing high requirements on the scale and quality of multimodal samples. These constraints significantly limit the performance of sensing intelligence in IoT applications, as the heterogeneity and the non-interpretability of time-series signals result in abundant unimodal data but scarce high-quality multimodal pairs. This paper proposes InfoMAE, a cross-modal alignment framework that tackles the challenge of multimodal pair efficiency under the SSL setting by facilitating efficient cross-modal alignment of pretrained unimodal representations. InfoMAE achieves efficient cross-modal alignment with limited data pairs through a novel information theory-inspired formulation that simultaneously addresses distribution-level and instance-level alignment. Extensive experiments on two real-world IoT applications are performed to evaluate InfoMAE's pairing efficiency to bridge pretrained unimodal models into a cohesive joint multimodal model. InfoMAE enhances downstream multimodal tasks by over 60% with significantly improved multimodal pairing efficiency. It also improves unimodal task accuracy by an average of 22%.

This paper introduces PhyMask, an adaptive masking paradigm designed to enhance the efficiency and interpretability of Masked Autoencoders (MAEs) in analyzing IoT sensing signals. Different from all mainstream MAEs, which rely on random masking techniques, PhyMask employs an adaptive masking strategy that aligns with critical signal information. Its main contributions are threefold. First, PhyMask leverages the energy significance of frequency components to prioritize information-rich time-frequency regions, improving the reconstruction of original signals. Second, it includes a coherence-based masking component to identify and preserve essential temporal dynamics within the data. Finally, PhyMask integrates these components into an adaptive masking paradigm tailored to optimize the sensing context awareness within the masking configuration, focusing on the most informative parts of the data. This allows PhyMask to mask up to 96% of the input, reducing memory requirements by 14% and accelerating pre-training. Evaluations across two sensing applications, four datasets, and two real-world deployments demonstrate PhyMask's superior performance. PhyMask improves MAE accuracy by 7%, reduces pre-training data requirements by up to 75%, and enhances robustness to domain shifts and signal quality variations, making it of great value to robust and efficient intelligent IoT deployments.

Video action recognition is a fundamental task for video understanding. Action recognition in complex spatio-temporal contexts generally requires fusing of different multi-granularity action information. However, existing works do not consider spatio-temporal information modeling and fusion from the perspective of action granularity. To address this problem, this paper proposes an Action Granularity Pyramid Network (AGPN) for action recognition, which can be flexibly integrated into 2D backbone networks. The core module is the Action Granularity Pyramid Module (AGPM), a hierarchical pyramid structure with residual connections, which is established to fuse multi-granularity action spatio-temporal information. From top to bottom level in the designed pyramid structure, the receptive field decreases and action granularity becomes more refined. To enrich temporal information of the inputs, a Multiple Frame Rate Module (MFM) is proposed to mix different frame rates at a fine-grained pixel-wise level. Moreover, a Spatio-temporal Anchor Module (SAM) is employed to fix spatio-temporal feature anchors to promote the effectiveness of feature extraction. We conduct extensive experiments on three large-scale action recognition datasets, Something-Something V1 & V2 and Kinetics-400. The results demonstrate that our proposed AGPN outperforms the state-of-the-art methods for the tasks of video action recognition.

In the past decade, with the rapid development of wireless communication and sensor technology, ubiquitous smartphones equipped with increasingly rich sensors have more powerful computing and sensing abilities. Thus, mobile crowdsensing has received extensive attentions from both industry and academia. Recently, plenty of mobile crowdsensing applications come forth, such as indoor positioning, environment monitoring, and transportation. However, most existing mobile crowdsensing systems lack vast user bases and thus urgently need appropriate incentive mechanisms to attract mobile users to guarantee the service quality. In this paper, we propose to incorporate sensing platform and social network applications, which already have large user bases to build a three-layer network model. Thus, we can publicize the sensing platform promptly in large scale and provide long-term guarantee of data sources. Based on a three-layer network model, we design incentive mechanisms for both intermediaries and the crowdsensing platform and provide a solution to cope with the problem of user overlapping among intermediaries. We theoretically prove the properties of our proposed incentive mechanisms, including incentive compatibility, individual rationality, and efficiency. Furthermore, we evaluate our incentive mechanisms by extensive simulations. Evaluation results validate the effectiveness and efficiency of our proposed mechanisms.

In the past decade, with the rapid development of wireless communication and sensor technology, ubiquitous smartphones equipped with increasingly rich sensors have more powerful computing and sensing abilities. Thus, mobile crowdsensing has received extensive attentions from both industry and academia. Recently, plenty of mobile crowdsensing applications come forth, such as indoor positioning, environment monitoring, transportation, and so on. However, most existing mobile crowdsensing systems lack of vast user bases, and thus urgently need appropriate incentive mechanisms to attract mobile users to guarantee the service quality. In this paper, we propose to incorporate sensing platform and social network applications, which already have large user bases to build a three-layer network model. Thus, we can publicize the sensing platform promptly in large scale, and provide longterm guarantee of data sources. Based on a three-layer network model, we design incentive mechanisms for both intermediaries and the crowdsensing platform, and provide a solution to cope with the problem of user overlapping among intermediaries. We indicate the properties of our proposed incentive mechanisms, including incentive compatibility, individual rationality, and efficiency.