Lane change prediction, encompassing both intention recognition and trajectory forecasting, is essential for the safe operation of autonomous vehicles in mixed-traffic environments. Existing models predominantly follow a data-driven paradigm, learning directly from historical vehicle states through an end-to-end approach. Inspired by the emerging paradigm of enhancing model generalizability through domain knowledge, we propose KnowLCP to explicitly model and integrate driving knowledge into the lane change prediction task. Specifically, we incorporate three types of knowledge: traffic risk awareness to improve intention prediction, vehicle kinematics to ensure the physical feasibility of predicted trajectories, and intention intensity to refine trajectory forecasting. Furthermore, we introduce a novel knowledge injection strategy that enhances mutual information during integration and proves superior to the traditional parallel input mechanism, which simply feeds knowledge features alongside historical states. Extensive experiments on two real-world trajectory datasets demonstrate that KnowLCP achieves average improvements of 8.3-10.3% in intention prediction and 10.1-10.3% in trajectory prediction over the best-performing baselines.

Vision Transformers (ViTs) have achieved strong performance in video action recognition, but their high computational cost limits their practicality. Lightweight CNNs are more efficient but suffer from accuracy gaps. Cross-Architecture Knowledge Distillation (CAKD) addresses this by transferring knowledge from ViTs to CNNs, yet existing methods often struggle with architectural mismatch and overlook the value of stronger homogeneous CNN teachers. To tackle these challenges, we propose a Dual-Teacher Knowledge Distillation framework that leverages both a heterogeneous ViT teacher and a homogeneous CNN teacher to collaboratively guide a lightweight CNN student. We introduce two key components: (1) Discrepancy-Aware Teacher Weighting, which dynamically fuses the predictions from ViT and CNN teachers by assigning adaptive weights based on teacher confidence and prediction discrepancy with the student, enabling more informative and effective supervision; and (2) a Structure Discrepancy-Aware Distillation strategy, where the student learns the residual features between ViT and CNN teachers via a lightweight auxiliary branch, focusing on transferable architectural differences without mimicking all of ViT’s high-dimensional patterns. Extensive experiments on benchmarks including HMDB51, EPIC-KITCHENS-100, and Kinetics-400, demonstrate that our method consistently outperforms state-of-the-art distillation approaches, achieving notable performance improvements with a maximum accuracy gain of 5.95% on HMDB51.

Recent advancements in multimodal recommendations, which leverage diverse modality information to mitigate data sparsity and improve recommendation accuracy, have gained significant attention. However, existing multimodal recommendations overlook the critical role of user representation initialization. Unlike items, which are naturally associated with rich modality information, users lack such inherent information. Consequently, item representations initialized based on meaningful modality information and user representations initialized randomly exhibit a significant semantic gap.

Federated Learning (FL) has emerged as a promising paradigm for decentralized machine learning, where a central server coordinates distributed clients to collaboratively train a global model without direct access to raw data. Despite its advantages, heterogeneous and long-tail data distributions across clients remain a major bottleneck, particularly in IoT scenarios with diverse devices and sensing modalities. To address these challenges, we propose FedSM, a novel framework that integrates multimodal semantic knowledge with balanced pseudo features to enhance global model optimization. Unlike conventional approaches that rely on single-modal information, FedSM leverages CLIP’s cross-modal representations and open-vocabulary priors to guide semantic-aware data augmentation. A probabilistic selection mechanism further refines local features by mixing them with global prototypes, ensuring pseudo features are semantically reliable and reducing bias caused by skewed client distributions. Almost all computations are performed locally at the client side, thereby alleviating server overhead and improving scalability in resource-constrained IoT environments. Extensive experiments on long-tail benchmarks including CIFAR-10-LT, CIFAR-100-LT, and ImageNet-LT demonstrate the superiority of FedSM over state-of-the-art baselines, highlighting its potential for robust communication-efficient FL in IoT networks.

Robust Reversible Watermarking (RRW) enables perfect recovery of cover images and watermarks in lossless channels while ensuring robust watermark extraction under lossy channels. However, existing RRW methods, mostly non-deep learning-based, suffer from complex designs, high computational costs, and poor robustness limiting their practical applications. To address these issues, this paper proposes Deep Robust Reversible Watermarking (DRRW), a deep learning-based RRW scheme. DRRW introduces an Integer Invertible Watermark Network (iIWN) to achieve an invertible mapping between integer data distributions, fundamentally addressing the limitations of conventional RRW approaches. Unlike traditional RRW methods requiring task-specific designs for different distortions, DRRW adopts an encoder-noise layer-decoder framework, enabling adaptive robustness against various distortions through end-to-end training. During inference, the cover image and watermark are mapped into an overflowed stego image and latent variables. Arithmetic coding efficiently compresses these into a compact bitstream, which is embedded via reversible data hiding to ensure lossless recovery of both the image and watermark. To reduce pixel overflow, we introduce an overflow penalty loss, significantly shortening the auxiliary bitstream while improving both robustness and stego image quality. Additionally, we propose an adaptive weight adjustment strategy that eliminates the need to manually preset the watermark loss weight, ensuring improved training stability and performance. Experiments on multiple datasets demonstrate that the proposed DRRW addresses key challenges in current RRW methods and significantly advances the practical deployment of RRW.

Facial Action Unit (AU) recognition involves identifying subtle muscle movements corresponding to different AUs. Recent approaches have focused on localizing AUs using predefined Regions of Interest (RoIs) or learnable modules. However, these methods either overly depend on the precision of predefined RoIs or inaccurately localize background regions instead of the actual AU positions. To address this challenge, we propose a novel method, which automatically localizes each AU without relying on predefined RoIs or introducing learnable modules during the inference phase. Specifically, our approach decomposes the task into two subtasks: AU localization and AU state verification. We first align the direction between spatial features and the corresponding AU class weights to guide the model in localizing AUs. Next, we incorporate spatial and temporal aspects for precise AU state detection. From the perspective of spatial information learning, we propose a confidence-based AU relationship mining module that directs the model to focus on uncertain AUs. From the aspect of temporal information learning, we introduce a temporal sampling strategy that implicitly captures time-dependent features. Experimental results on the BP4D and DISFA datasets demonstrate the effectiveness of our method, showing that it outperforms existing approaches and achieves state-of-the-art performance in AU recognition.

Training neural radiance fields without pre-estimated camera poses remains challenging, particularly in the presence of complex illumination and large inter-frame motion. This work presents a self-calibrating radiance-field framework designed to address these difficulties through a unified optimization strategy. Reliable monocular depth priors from the Depth Anything V2 model are incorporated to provide stable geometric constraints, and multiview point-cloud alignment is enforced to prevent pose drift under wide baselines. Illumination-induced photometric inconsistencies are handled by a jointly optimized exposure-compensation module that separates transient brightness variations from the underlying scene radiance. To enhance the representational capacity beyond that of standard multilayer perceptrons, a Kolmogorov-Arnold Network is adopted as the radiance-field backbone, enabling improved recovery of fine geometric and appearance details. The complete pipeline is deployed on a cloud-native infrastructure that supports scalable optimization and interactive access through an LLM-assisted interface. Experiments on indoor and outdoor datasets demonstrate that the proposed method consistently improves pose accuracy and novel-view synthesis quality over existing self-calibrating approaches.

Currently, the prevailing approach in voice conversion (VC) involves separating clearer linguistic information from the source audio and then reconstructing it with the identity of the target speaker. However, existing methods, whether employing in-formation perturbation techniques or carefully designed information bottleneck methods, encounter challenges related to unsatisfactory audio separation effects and insufficient robustness. This article introduces a VC through the self-supervised learning method (SSL-VC). First, it utilizes a self-supervised speech representation (S-SSR) extraction network with decoupling (Decp-SSEN) to disentangle linguistic information from speech. The designed prosodic encoder extracts features of pitch and energy from the speech to compensate for the loss of nonlinguistic details incurred during the Decp-SSEN disentangling process. This approach allows us to obtain richer linguistic information independent of speaker identity, guaranteeing the robust performance of the model. Second, we leverage high-level S-SSR as the intermediate feature, replacing the traditional Mel-spectrogram. Built an end-to-end VC pipeline that eliminates the need for a vocoder, enhancing the expression level of intermediate features and reducing the learning difficulty gap between real and predicted features. Subjective and objective experiments conducted on both seen and unseen speech corpus demonstrate that SSL-VC achieves high-quality VC and speaker similarity. Moreover, it outperforms state-of-the-art methods in extracting richer linguistic information. Ablation experiments further scrutinize the indispensability of the prosodic encoder.

Zero-shot temporal action localization (ZSTAL) aims to localize and recognize action categories unseen during training. However, it assumes that test videos contain only unseen classes, which is unrealistic in practice where seen and unseen actions naturally co-exist. To bridge this gap, we introduce generalized ZSTAL (GZS-TAL), where models trained only on seen classes must handle both seen and unseen ones during testing. This setting highlights a critical challenge: a static, frozen model cannot adapt to the mixed distributions encountered at test time. To address this issue, we propose a Temporal-Sensitive Adaptation (TSA) module that equips TAL models with the ability to update themselves during testing. The key intuition is to use temporal dependency prediction as a self-supervised signal: TSA introduces an online-updatable memory optimized to reconstruct features of preceding segments from the current one, thereby embedding temporal dependencies into parameters and reusing them for adaptation at test time. To further enhance temporal modeling, we extend TSA into a Bi-directional TSA (Bi-TSA) mechanism that performs prediction in both forward and backward directions. By simultaneously exploiting historical and future contexts, Bi-TSA improves long-range temporal representation and yields more accurate boundary localization. Extensive experiments on THUMOS14 and ActivityNet-1.3 demonstrate that our approach achieves significant improvements over state-of-the-art methods under the GZS-TAL setting, validating its effectiveness and generalization ability.

Despite continuous advancements in deep learning for understanding human motion, existing models often struggle to accurately identify action timing and specific body parts, typically supporting only single-round interaction. This limitation is particularly pronounced in home exercise monitoring, neurological disorder assessment, and rehabilitation, where precise motion analysis is crucial for ensuring exercise efficacy, detecting early signs of neurological conditions, and guiding personalized recovery programs. In this paper, we propose MoChat, a multimodal large language model capable of spatio-temporal grounding of human motion and multi-turn dialogue understanding. To achieve this, we first group spatial features in skeleton frames according to human anatomical structures and process them through a Joints-Grouped Skeleton Encoder. The encoder's outputs are fused with large language model embeddings to generate spatio-aware representations. A cross-attention-based Regression Head module is then designed to align hidden-layer embeddings and skeletal sequence embeddings, enabling precise temporal grounding. Furthermore, we develop a pipeline for temporal grounding task to extract timestamps from skeleton-text pairs and construct a multi-turn instruction dialogues for spatial grounding task. Finally, various task instructions are generated for jointly training. Experimental results demonstrate that MoChat achieves state-of-the-art performance across multiple metrics in motion understanding tasks, making it as the first model capable of fine-grained spatio-temporal grounding of human motion.

Electrocardiography (ECG) is the gold standard for monitoring heart function and is crucial for preventing the worsening of cardiovascular diseases (CVDs). However, the inconvenience of ECG acquisition poses challenges for long-term continuous monitoring. Consequently, researchers have explored non-invasive and easily accessible photoplethysmography (PPG) as an alternative, converting it into ECG. Previous studies have focused on peaks or simple mapping to generate ECG, ignoring the inherent periodicity of cardiovascular signals. This results in an inability to accurately extract physiological information during the cycle, thus compromising the generated ECG signals' clinical utility. To this end, we introduce a novel PPG-to-ECG translation model called CATransformer, capable of adaptive modeling based on the cardiac cycle. Specifically, CATransformer automatically extracts the cycle using a cycle-aware module and creates multiple semantic views of the cardiac cycle. It leverages a transformer to capture detailed features within each cycle and the dynamics across cycles. Our method outperforms existing approaches, exhibiting the lowest RMSE across five paired PPG-ECG databases. Additionally, extensive experiments are conducted on four cardiovascular-related tasks to assess the clinical utility of the generated ECG, achieving consistent state-of-the-art performance. Experimental results confirm that CATransformer generates highly faithful ECG signals while preserving their physiological characteristics.

Federated learning (FL) is vulnerable to gradient-based privacy attacks, where malicious attackers reconstruct training data from exchanged gradients. While existing differential privacy (DP) defenses mitigate this, they often cause excessive additive noise due to the inequality scaling in the theoretical analyses, which degrades the model's utility or fail under adaptive attacks. To address this issue, we proposeFedMSBA, a layer-wise privacy-preservation method that adaptively allocates privacy budgets via Rényi DP (RDP) and modified sensitivity. FedMSBA dynamically scales noise to model intricacies and adaptively choose the better applied DP mechanisms, which provides a tighter mathematical bound and finally prevents non-convergence while resisting reconstruction attacks. Experiments demonstrate superior privacy-utility trade-offs compared to state-of-the-art defenses. FedMSBA achieves an approximately 2% improvement in accuracy and a 5% enhancement in privacy preservation. Furthermore, FedMSBA's performance remains nearly unaffected by variations in the privacy budget$\epsilon$and failure rate$\delta$.

The exponential growth of online information has made it increasingly difficult for users to identify valuable and relevant content. Recommender systems have emerged as a critical solution to this challenge by tailoring content to individual preferences. With the proliferation of diverse multimedia services, human interaction with the digital world has become inherently multimodal. Consequently, recommender systems capable of comprehending and interpreting multimodal information can more effectively align with individual preferences. With its recent surge in research attention, the field of multimodal recommender systems (MRS) still lacks a comprehensive technical survey. Existing surveys suffer from two critical limitations: 1) Insufficient technical depth: Prior works predominantly focus on categorizing and discussing general structure, neglecting rigorous technical analysis of methodologies and architectures. 2) Absence of cutting-edge works: Due to the rapid evolution of AI technologies, current surveys fail to discuss the latest works that adopt the most advanced techniques. To bridge these gaps, this survey conducts a systematic and technical review of advanced MRS works from MRS's inception to the present. We organize existing works into coherent taxonomies based on their structure and provide in-depth analyses of methodological innovations at each component, including Feature Extraction, Encoder, Multimodal Fusion, and Loss Function. Moreover, we further discuss potential future directions for developing and enhancing MRS. This survey serves as technical guidance for researchers and practitioners, offering insights into the developments, techniques, and future directions of MRS. Notably, we maintain a public repository:https://github.com/Jinfeng-Xu/Awesome-Multimodal-Recommender-Systems.

Beyond traditional user recommendation, group recommendation is a new and popular task that provides recommendations for a group of users. Previous works aggregate member preferences in the group to infer group preference, but this often leads to a coarse-grained inference for group preferences limited by users’ individual preferences. To this end, we exploit that user preferences can be inferred and refined by exploring the group preferences that they participated in. These refined preferences offer additional information beyond the original individual preferences, enabling more fine-grained and satisfactory group preference inference. In this work, we propose a novel Dual-Granularity Graph Variational Auto-Encoder framework (DGGVAE) for group recommendation, which jointly reveals group preferences from both coarse granularity and fine granularity to comprehensively learn group preferences. Specifically, we design a Group Preference Extractor module that extracts group preferences from these two granularities: coarse granularity, which is revealed through original member preferences, and fine granularity, which is revealed through refined member preferences. To extract the correlation between groups, a Group Representation Enhancement module is proposed, which enhances group representations by information from the most similar groups. However, the coarse- and fine-grained group preferences contain uncertainty due to the gap between the original and refined member preferences. To better incorporate dual-granularity group preferences, we design granularity-specific graph variational encoders that learn Gaussian variables on the semantic information for each group. Moreover, with the conditional independence assumption, the granularity-specific Gaussian node embeddings are fused according to the generalized product-of-experts (gPoE), where the semantic information in each granularity is weighted based on the estimated uncertainty level. Extensive experiments show the superiority of DGGVAE over various state-of-the-art methods in training efficiency and accuracy on both group and user recommendation tasks.

With the rapid evolution of the Decentralized Web (DWeb), decentralized technologies have paved new avenues for Web3 applications and the authentication of digital assets. Among them, Non-Fungible Tokens (NFTs) have gained significant popularity due to their immutability and uniqueness, reshaping the landscape of artistic creation, marketing, and intellectual property protection. However, current blockchain-based NFT implementations still face core challenges within decentralized architecture: how to maintain decentralization while ensuring the visual uniqueness of digital assets and reducing storage costs. The rampant issue of duplication undermines the scarcity of digital art and erodes market confidence in copyright authenticity. Moreover, high gas fees and energy consumption further hinder the widespread adoption of NFTs, while reliance on external storage solutions like InterPlanetary File System (IPFS) introduces risks of data instability and loss. To address these challenges, this article presents the UniqueNFT framework, a novel architecture that deeply integrates blockchain oracles with decentralized storage verification mechanisms. The framework achieves three key technological breakthroughs: Using image inversion and generation techniques based on Encoder for Editing (E4E) and StyleGAN3, it extracts compact and expressive semantic features from NFT images, enabling efficient data compression and significantly reducing on-chain storage volume; The Crypto-Mask algorithm, by utilizing the hash value of blockchain user information (user-controlled SHA-256 digest of Ethereum address, user nickname, and registration time), ensures the visual uniqueness of NFTs; A smart contract extension compatible with the ERC721 standard, demonstrating UniqueNFT’s seamless integration within the blockchain ecosystem. By leveraging the technologies of the Decentralized Web, our framework represents an important step forward in enhancing the security and uniqueness of digital assets. It not only innovatively resolves the issues of NFT duplication and homogenization but also injects new vitality and long-term momentum into the creation of a trusted, sustainable blockchain-based digital asset ecosystem.

Test-time adaptation (TTA) aims to fine-tune a trained model online using unlabeled testing data to adapt to new environments or out-of-distribution data, demonstrating broad application potential in real-world scenarios. However, in this optimization process, unsupervised learning objectives like entropy minimization frequently encounter noisy learning signals. These signals produce unreliable gradients, which hinder the model’s ability to converge to an optimal solution quickly and introduce significant instability into the optimization process. In this paper, we seek to resolve these issues from the perspective of optimizer design. Unlike prior TTA using manually designed optimizers like SGD, we employ a learning-to-optimize approach to automatically learn an optimizer, called Meta Gradient Generator (MGG). Specifically, we aim for MGG to effectively utilize historical gradient information during the online optimization process to optimize the current model. To this end, in MGG, we design a lightweight and efficient sequence modeling layer -- gradient memory layer. It exploits a self-supervised reconstruction loss to compress historical gradient information into network parameters, thereby enabling better memorization ability over a long-term adaptation process. We only need a small number of unlabeled samples to pre-train MGG, and then the trained MGG can be deployed to process unseen samples. Promising results on ImageNet-C/R/Sketch/A indicate that our method surpasses current state-of-the-art methods with fewer updates, less data, and significantly shorter adaptation times. Compared with a previous SOTA SAR, we achieve 7.4% accuracy improvement and 4.2x faster adaptation speed on ImageNet-C.

Emotion recognition based on body movements is vital in human-computer interaction. However, existing emotion recognition methods predominantly focus on enhancing classification accuracy, often neglecting the provision of textual explanations to justify their classifications. In this paper, we propose an Emotion-Action Interpreter powered by LargeLanguage Model (EAI-LLM), which not only recognizes emotions but also generates textual explanations by treating 3D body movement data as unique input tokens within large language models (LLMs). Specifically, we propose a multi-granularity skeleton tokenizer designed for LLMs, which separately extracts spatio-temporal tokens and semantic tokens from the skeleton data. This approach allows LLMs to generate more nuanced classification descriptions while maintaining robust classification performance. Furthermore, we treat the skeleton sequence as a specific language and propose a unified skeleton token module. This module leverages the extensive background knowledge and language processing capabilities of LLMs to address the challenges of joint training on heterogeneous datasets, thereby significantly enhancing recognition accuracy on individual datasets. Experimental results demonstrate that our model achieves recognition accuracy comparable to existing methods. More importantly, with the support of background knowledge from LLMs, our model can generate detailed emotion descriptions based on classification results, even when trained on a limited amount of labeled skeleton data.

The success of Large Language Models (LLMs) relies heavily on the huge amount of pre-training data learned in the pre-training phase. The opacity of the pre-training process and the training data causes the results of many benchmark tests to become unreliable. If any model has been trained on a benchmark test set, it can seriously hinder the health of the field. In order to automate and efficiently test the capabilities of large language models, numerous mainstream benchmarks adopt a multiple-choice format. As the swapping of the contents of multiple-choice options does not affect the meaning of the question itself, we propose a simple and effective data leakage detection method based on this property. Specifically, we shuffle the contents of the options in the data to generate the corresponding derived data sets, and then detect data leakage based on the model's log probability distribution over the derived data sets. If there is a maximum and outlier in the set of log probabilities, it indicates that the data is leaked. Our method is able to work under gray-box conditions without access to model training data or weights, effectively identifying data leakage from benchmark test sets in model pre-training data, including both normal scenarios and complex scenarios where options may have been shuffled intentionally or unintentionally. Through experiments based on two LLMs and benchmark designs, we demonstrate the effectiveness of our method. In addition, we evaluate the degree of data leakage of 35 mainstream open-source LLMs on four benchmark datasets and give a ranking of the leaked LLMs for each benchmark, and we find that the Qwen family of LLMs has the highest degree of data leakage.

Graph Collaborative Filtering (GCF) has emerged as a dominant paradigm in modern recommendation systems, excelling at modeling complex user-item interactions and capturing high-order collaborative signals. Most existing GCF models predominantly rely on simplified graph architectures like LightGCN, which strategically remove feature transformation and activation functions from vanilla graph convolution networks. Through systematic analysis, we reveal that feature transformation in message propagation can enhance model representation, though at the cost of increased training difficulty. To this end, we propose FourierKAN-GCF, a novel framework that adopts Fourier Kolmogorov-Arnold Networks as efficient transformation modules within graph propagation layers. This design enhances model representation while decreasing training difficulty. Our FourierKAN-GCF can achieve higher recommendation performance than most widely used GCF backbone models and can be integrated into existing advanced self-supervised models as a backbone, replacing their original backbone to achieve enhanced performance. Extensive experiments on three public datasets demonstrate the superiority of FourierKAN-GCF.

Soft continuum robots have broad application prospects due to their superior flexibility, compliance and safety. However, the high mechanical complexity addresses a major challenge in precise pose control modeling. Deep Deterministic Policy Gradient (DDPG) algorithm, as a foundational modelfree, off-policy method, is well-suited for such tasks. However, DDPG is known for its inherent instabilities, particularly its susceptibility to Q-value overestimation and challenges in sparse reward environments, which can hinder performance and efficiency. To address these limitations, this paper proposes a Hybrid Reward-Shaped DDPG and PSO fusion algorithm(HR-DP). This method integrates a multi-component reward function with the PSO algorithm, aiming to guide the agent's exploration more effectively and higher exploration accuracy. The target point tracking experiment by the two-segment soft continuum robot in simulation was performed. The results demonstrate that the robot controlled by HR-DP can achieve significantly faster convergence, higher cumulative rewards, higher accuracy and greater stability than that by the standard DDPG baseline and other state-of-theart methods. This work provides a practical and effective HR-DP algorithm enhanced from DDPG, which can benefit pose control of the multi-segment soft continuum robot in complex, real-world environment.

To address the flexible operation requirements of narrow workspace in minimally invasive surgery(MIS), this paper proposes a continuum robot system featuring cooperative traction-torsion actuation via flexible shafts. The flexible sections realize Omni-bendable movements by traction of the flexible shafts, while the rigid section achieves fine positioning through gear transmission driven by the rotary of the flexible shafts. In order to formulate the relation between the distal position of the continuum robot and the traction of the flexible shafts, a dynamics model is established in Adams to generate simulation data. A nonlinear mapping between the end-effector position and multi-flexible-shaft inputs is constructed by training the simulated data in a back-propagation(BP) neural network. O-shaped and Sshaped trajectory-tracking and pose-comparison tests are conducted in Adams-Matlab/Simulink co-simulation environment. The results demonstrates that the end effector of the continuum robot can achieve high positioning accuracy and stable tracking, validating the feasibility and potential application in MIS.

The confine workspace in minimally invasive surgery and the difficulty of installing force sensor at the distal of surgical instrument make it necessary for surgeons to rely on clinical experience to assess the safety of the surgical site. This paper proposes a force-sensor-free method for perceiving gripping force in the opening-and-closing mechanism of a surgical manipulator driven by a flexible shaft. By constructing a dataset between the rotational torque of the flexible shaft and the clamping force of the opening-and-closing mechanism, an end-toend mapping model between flexible shaft torque and clamping force is established by incorporating the dataset into a multi-layer perceptron(MLP) neural network model,. This enables the acquisition of the clamping force exerted by the surgical instrument on the target tissue without relying on pressure sensors at the instrument tip, thereby enhancing the immersion and safety of surgical procedures.

In this paper, we introduce a novel psychological benchmark, CPsyExam, constructed from questions sourced from Chinese examination systems. CPsyExam is designed to prioritize psychological knowledge and case analysis separately, recognizing the significance of applying psychological knowledge to real-world scenarios. We collect 22k questions from 39 psychology-related subjects across four Chinese examination systems. From the pool of 22k questions, we utilize 4k to create the benchmark that offers balanced coverage of subjects and incorporates a diverse range of case analysis techniques. Furthermore, we evaluate a range of existing large language models (LLMs), spanning from open-sourced to proprietary models. Our experiments and analysis demonstrate that CPsyExam serves as an effective benchmark for enhancing the understanding of psychology within LLMs and enables the comparison of LLMs across various granularities.

Temporal action detection (TAD) aims to identify and localize action instances in untrimmed videos, which is essential for various video understanding tasks. However, recent improvements in model performance, driven by larger feature extractors and datasets, have led to increased computational demands. This presents a challenge for applications like autonomous driving and robotics, which rely on limited computational resources. While existing channel pruning methods can compress these models, reducing the number of channels often hinders the parallelization efficiency of GPU, due to the inefficient multiplication between small matrices. Instead of pruning channels, we propose a Progressive Block Drop method that reduces model depth while retaining layer width. In this way, we still use large matrices for computation but reduce the number of multiplications. Our approach iteratively removes redundant blocks in two steps: first, we drop blocks with minimal impact on model performance; and second, we employ a parameter-efficient cross-depth alignment technique, fine-tuning the pruned model to restore model accuracy. Our method achieves a 25% reduction in computational overhead on two TAD benchmarks (THUMOS14 and ActivityNet-1.3) to achieve lossless compression. More critically, we empirically show that our method is orthogonal to channel pruning methods and can be combined with it to yield further efficiency gains.

With the extraordinary success of generative artificial intelligence, large pretrained models (LPMs) have been widely used to achieve human-level performance. Despite the one-shot capability, it is always preferred to fine-tune the LPMs for domain-specific downstream tasks. Therefore, the federated learning system is leveraged to fine-tune the large pretrained models enabling concurrrently use multiple distributed clients as well as their local datasets. While the first-order fine-tuning methods suffer from high computational and memory costs due to the backward propagation, we are motivated to propose a federated zeroth-order fine-tuning method with only forward propagation. Moreover, we also leverage differential privacy to further preserve the data privacy of local clients. Experimental results illustrate that our proposed federated zeroth-order method can reduce the memory and retain a similar testing accuracy over the state-of-the-art benchmarks.

Video grounding (VG) task focuses on locating specific moments in a video based on a query, usually in text form. However, traditional VG struggles with some scenarios like streaming video or queries using visual cues. To fill this gap, we present a new task named Online Video Grounding with Hybrid-modal Queries (OVG-HQ), which enables online segment localization using text, images, video segments, and their combinations. This task poses two new challenges: limited context in online settings and modality imbalance during training, where dominant modalities overshadow weaker ones. To address these, we propose OVG-HQ-Unify, a unified framework featuring a Parametric Memory Block (PMB) that retain previously learned knowledge to enhance current decision and a cross-modal distillation strategy that guides the learning of non-dominant modalities. This design enables a single model to effectively handle hybrid-modal queries. Due to the lack of suitable datasets, we construct QVHighlights-Unify, an expanded dataset with multi-modal queries. Besides, since offline metrics overlook prediction timeliness, we adapt them to the online setting, introducing oR@n, IoU=m, and online mean Average Precision (omAP) to evaluate both accuracy and efficiency. Experiments show that our OVG-HQ-Unify outperforms existing models, offering a robust solution for online, hybrid-modal video grounding. Source code and datasets are available at https://github.com/maojiaqi2324/OVG-HQ.

Federated learning (FL) enables collaborative training of a global model in the centralized server with data from multiple parties while preserving privacy. However, data heterogeneity can significantly degrade the performance of the global model when each party uses datasets from different sources to train a local model, thereby affecting personalized local models. Among various cases of data heterogeneity, feature drift, feature space difference among parties, is prevalent in real-life data but remains largely unexplored. Feature drift can distract feature extraction learning in clients and thus lead to poor feature extraction and classification performance. To tackle the problem of feature drift in FL, we propose FedPall, an FL framework that utilizes prototype-based adversarial learning to unify feature spaces and collaborative learning to reinforce class information within the features. Moreover, FedPall leverages mixed features generated from global prototypes and local features to enhance the global classifier with classification-relevant information from a global perspective. Evaluation results on three representative feature-drifted datasets demonstrate FedPall's consistently superior performance in classification with feature-drifted data in the FL scenario.

Electrocardiogram (ECG) is widely used to diagnose cardiac conditions via deep learning methods. Although existing self-supervised learning (SSL) methods have achieved great performance in learning representation for ECG-based cardiac conditions classification, the clinical semantics can not be effectively captured. To overcome this limitation, we proposed to learn cross-modal ECG representations that contain more clinical semantics via a novel framework with \textbf{D}eep \textbf{E}CG-\textbf{R}eport \textbf{I}nteraction (\textbf{DERI}). Specifically, we design a novel framework combining multiple alignments and mutual feature reconstructions to learn effective representation of the ECG with the clinical report, which fuses the clinical semantics of the report. An RME-Module inspired by masked modeling is proposed to improve the ECG representation learning. Furthermore, we extend ECG representation learning to report generation with a language model, which is significant for evaluating clinical semantics in the learned representations and even clinical applications. Comprehensive experiments with various settings are conducted on various datasets to show the superior performance of our DERI. Our code is released on https://github.com/cccccj-03/DERI.

Self-supervised learning (SSL) has garnered increasing attention in electrocardiogram (ECG) analysis for its effectiveness in resource-limited settings. Existing state-of-the-art SSL methods rely on time-frequency detail reconstruction, but due to the inherent redundancy of ECG signals and individual variability, these approaches often yield suboptimal performance. In contrast, discrete label prediction becomes a superior pre-training objective by encouraging models to efficiently abstract ECG high-level semantics. However, the continuity and significant variability of ECG signals pose a challenge in generating semantically discrete labels. To address this issue, we propose an ECG pretraining framework with a self-distillation semantic tokenizer (ECG2TOK), which maps continuous ECG signals into discrete labels for self-supervised training. Specifically, the tokenizer extracts semantically aware embeddings of ECG by self-distillation and performs online clustering to generate semantically rich discrete labels. Subsequently, the SSL model is trained in conjunction with masking strategies and discrete label prediction to facilitate the abstraction of high-level semantic representations. We evaluate ECG2TOK in six downstream tasks, demonstrating that ECG2TOK efficiently achieves state-of-the-art performance and up to a 30.73% AUC increase in low-resource scenarios. Moreover, visualization experiments demonstrate that the discrete labels generated by ECG2TOK exhibit consistent semantics closely associated with clinical features. Our code is available on https://github.com/YXYanova/ECG2TOK.

Personalized Federated Learning (PFL) targets client-specific models under heterogeneous and limited data. However, conventional methods often use heuristic or data-size-based averaging and overlook the true contributions of client updates. We propose a contribution-oriented PFL framework that quantifies client contributions via gradient alignment and prediction discrepancy for informed aggregation. We further develop a parameter-wise personalization mechanism for adaptive local updates and a mask-aware momentum optimizer for stable training. Preliminary results on CIFAR10 validate its effectiveness.

Recent works in multimodal recommendations, which leverage diverse modal information to address data sparsity and enhance recommendation accuracy, have garnered considerable interest. Two key processes in multimodal recommendations are modality fusion and representation learning. Previous approaches in modality fusion often employ simplistic attentive or pre-defined strategies at early or late stages, failing to effectively handle irrelevant information among modalities. In representation learning, prior research has constructed heterogeneous and homogeneous graph structures encapsulating user-item, user-user, and item-item relationships to better capture user interests and item profiles. Modality fusion and representation learning were considered as two independent processes in previous work. This paper reveals that these two processes are complementary and can support each other. Specifically, powerful representation learning enhances modality fusion, while effective fusion improves representation quality. Stemming from these two processes, we introduce a COmposite grapH convolutional nEtwork with dual-stage fuSION for the multimodal recommendation, named COHESION. Specifically, it introduces a dual-stage fusion strategy to reduce the impact of irrelevant information, refining all modalities using behavior modality in the early stage and fusing their representations at the late stage. It also proposes a composite graph convolutional network that utilizes user-item, user-user, and item-item graphs to extract heterogeneous and homogeneous latent relationships within users and items. Besides, it introduces a novel adaptive optimization to ensure balanced and reasonable representations across modalities. Extensive experiments on three public datasets demonstrate the significant superiority of COHESION over various competitive baselines.

Recent advances in Artificial Intelligence (AI) have introduced a popular paradigm in Machine Learning (ML) model development: pre-training and domain adaptation. As both closed-source developers and open-source community lead in pre-training foundation models, domain deployers face the dilemma about whether to use closed-source models via API access or to host open-source models on proprietary hardware. Using closed-source models incurs recurring costs, while hosting open-source models requires substantial hardware investments and may lead to potentially lagging advancements. This paper presents a game-theoretical model to examine the economic incentives behind the deployment choice and the impact of open-source engagement strategies on technological innovation. We find that deployers consistently opt for closed-source APIs when the open-source community engages reactively by maintaining a fixed performance ratio relative to closed-source advancements. However, open-source models can become preferable when a proactive open-source community produces high-performance models independently. Furthermore, we identify conditions under which the engagement and competitiveness of the open-source community can either foster or inhibit technological progress. These insights offer valuable implications for market regulation and the future of technology innovation.

In recent years, federated learning (FL) has continued to evolve with the advent of big data and the large language model (LLM), but it has also exposed numerous security and privacy issues. As a form of distributed machine learning, FL systems are more susceptible to poisoning attacks because training data are dispersed across different participants; additionally, the training achievement of FL may be subject to low-cost theft by some free-riders. Existing works have addressed defenses against the aforementioned two types of threats, but they often focus on defending against only one type and fail to effectively integrate defenses against multiple types of threats. However, in real-world Internet of Things (IoT) systems, the types of threats are not limited to just one category. In this work, we try to maintain the performance of the global model under poisoning attacks, preserve the privacy of the server under free-riders, and explore the balance between these two aspects. Therefore, this work proposes Federated Multi-Stage Asynchronous Roll-back (FedMAR), ensuring the quality of local updates; in addition, this work also provides privacy preservation in the global update process based on Rinyi Differential Privacy (RDP), and offers a certain basis for detecting free-riders. To validate the generalization of the proposed method, we conducted relevant experiments on both image and text datasets, and further investigated the robustness of the proposed method against poisoning attacks, model inversion attacks, data heterogeneity, and other aspects. The testing accuracy of the global model can even be improved by 7.2%.

Federated learning (FL) emerged as a machine learning approach in situations where the privacy of sensitive data needs to be protected. Within the FL framework, clients collaborate to train a shared global model using their individual data by sending the model parameters to a central server, all while keeping their private data localized. Albeit its advantage, FL still faces certain limitations, such as clients may lack the motivation to participate in training, and the heterogeneous data distribution among clients can slow down the model convergence rate and degrade the model accuracy. In the light of the above mentioned considerations, we introduce entropy-based clustered FL (EntroCFL) with incentive mechanism, a two-layer clustered FL (CFL) model to jointly address the incentive mechanism and model training performance issues with heterogeneous clients. In Layer I, the server designs the payments to the clients to minimize its cost, including the training accuracy loss and the payment to clients, based on which the clients determine their training datasizes to maximize their own utilities. In Layer II, we introduce an entropy-based clustering method that is implemented based on the clients’ strategies in Layer I. Unlike conventional CFL methods that rely solely on the cosine similarity between clients’ parameter gradients, EntroCFL introduces a novel clustering discriminant which takes both angle and magnitude of clients’ parameter gradients into consideration. Simulation experiments are conducted to compare EntroCFL with conventional methods, such as FedAvg on the MNIST, EMNIST, and FMNIST datasets. The results validate the superiority of EntroCFL in terms of experimental accuracy, robustness, and economic efficiency.

To address the growing challenge of energy efficiency in next-generation coordinated multipoint (CoMP) communication systems, this article develops a green CoMP optimization framework that integrates renewable energy harvesting, smart grid interactions, and real-time power control. We formulate a stochastic long-term weighted sum-rate maximization problem, incorporating transmit covariance variables and joint channel-aware precoding. To enable online implementation, we convert the time-averaged problem into an equivalent per-slot formulation and design an online dynamic beamforming and energy management (ODBEM) algorithm. The proposed ODBEM integrates three synergistic mechanisms: 1) dual-driven energy pricing; 2) Lyapunov drift-plus-penalty scheduling; and 3) momentum-based energy smoothing. We further conduct rigorous convexity and Karush–Kuhn–Tucker optimality analysis to ensure algorithmic correctness and convergence. Simulation results demonstrate that ODBEM outperforms baseline strategies in both throughput and energy cost, confirming its effectiveness for sustainable and adaptive CoMP transmission.

With the rise of the Internet of Things, wearable devices have enabled real-time health monitoring, particularly through physiological signals like electrocardiograms (ECG). The standard 12-lead ECG records the electrical activity of the heart from multiple perspectives, providing valuable insights into cardiac health. However, existing 12-lead ECG analysis methods often treat leads as channel-level arrangements or rely on spatial adjacency to predefine lead connections, limiting their ability to capture the complex spatial and functional relationships between leads fully. To address this limitation, we propose TGLLNet, a task-driven model that automatically learns interlead relationships to improve multilabel ECG classification. TGLLNet adaptively learns lead connectivity patterns and relational strengths, enhancing ECG representation and improving model generalizability across tasks. Specifically, TGLLNet employs a temporal graph construction module to convert ecg signals into temporal graphs and uses a residual pyramid graph convolution module for multilevel graph embeddings, utilizing a graph convolutional network with independently learnable adjacency matrices. Combined with a temporal context convolution module, TGLLNet captures spatio-temporal dependencies, significantly improving ECG representation. Experimental results on seven tasks from PTB-XL and CPSC2018 datasets demonstrate that TGLLNet outperforms existing methods, showing superior generalizability across different tasks. Our code is available athttps://github.com/rosemary333/TGLLnet.

Self-supervised emotion recognition leveraging skeleton-based data offers a promising approach for classifying emotional expressions within the extensive amount of unlabeled data gathered by sensors in the Internet of Things (IoT). Recent advancements in this field have been driven by contrastive learning-based or generative learning-based self-supervised methods, which effectively tackle the issue of sparsely labeled data. In emotion recognition tasks, the emotional high-level semantics embedded in the skeleton data are more important than the subtle joint movements. Compared to existing methods, discrete label prediction can encourage SSL models to abstract high-level semantics in a manner similar to human perception. However, it is challenging to comprehensively capture emotional expressed in skeleton data solely from joint-based features. Moreover, emotional information conveyed through body movements may include redundant details that hinder the understanding of emotional expression. To overcome these challenges, we propose a novel discrete-label-based emotion recognition framework named the Appendage-Informed Redundancy-ignoring (AIR) discrete label framework. First, we introduce the Appendage-Skeleton Partitioning (ASP) module, which leverages limb movement data from the original skeleton to explore emotional expression. Next, we propose the Appendage-refined Multi-scale Discrete Label (AMDL) module, which transforms traditional self-supervised tasks into classification tasks. This design continuously extracts emotional semantics from skeleton data during pre-training, functioning similarly to predicting categories and subsequently classifying samples. To further reduce the nonessential information in skeleton data that may negatively impact the generation of accurate emotional categories, we propose the Appendage Label Refinement (ALR) module. It refines the generated categories by using the relationships between the skeleton and the various appendages obtained via ASP module. Finally, to maintain consistency across multiple scales, we introduce the Multi-Granularity Appendage Alignment (MGAA) method. By incorporating features from both coarse and fine scales, MGAA mitigates the encoder’s sensitivity to noise and enhances its overall robustness. We evaluate our approach on the Emilya, EGBM, and KDAE datasets, where it consistently outperforms state-of-the-art methods under various evaluation protocols.

Federated learning (FL), recognized for its decentralized and privacy-preserving nature, faces vulnerabilities to backdoor attacks that aim to manipulate the model's behavior on attacker-chosen inputs. Most existing defenses based on statistical differences take effect only against specific attacks. This limitation becomes significantly pronounced when malicious gradients closely resemble benign ones or the data exhibits non-IID characteristics, making the defenses ineffective against stealthy attacks. This paper revisits distance-based defense methods and uncovers two critical insights: First, Euclidean distance becomes meaningless in high dimensions. Second, a single metric cannot identify malicious gradients with diverse characteristics. As a remedy, we propose FedID, a simple yet effective strategy employing multiple metrics with dynamic weighting for adaptive backdoor detection. Besides, we present a modified z-score approach to select the gradients for aggregation. Notably, FedID does not rely on predefined assumptions about attack settings or data distributions and minimally impacts benign performance. We conduct extensive experiments on various datasets and attack scenarios to assess its effectiveness. FedID consistently outperforms previous defenses, particularly excelling in challenging Edge-case PGD scenarios. Our experiments highlight its robustness against adaptive attacks tailored to break the proposed defense and adaptability to a wide range of non-IID data distributions without compromising benign performance.

Trajectory prediction is a crucial challenge in autonomous vehicle motion planning and decision-making techniques. However, existing methods face limitations in accurately capturing vehicle dynamics and interactions. To address this issue, this article proposes a novel approach to extracting vehicle velocity and acceleration, enabling the learning of vehicle dynamics and encoding them as auxiliary information. The VDI-LSTM model is designed, incorporating graph convolution and attention mechanisms to capture vehicle interactions using trajectory data and dynamic information. Specifically, a dynamics encoder is designed to capture the dynamic information, a dynamic graph is employed to represent vehicle interactions, and an attention mechanism is introduced to enhance the performance of LSTM and graph convolution. To demonstrate the effectiveness of our model, extensive experiments are conducted, including comparisons with several baselines and ablation studies on real-world highway datasets. Experimental results show that VDI-LSTM outperforms other baselines compared, which obtains a 3% improvement on the average RMSE indicator over the five prediction steps.

Test-time adaptation (TTA) aims to boost the generalization capability of a trained model by conducting self-/unsupervised learning during the testing phase. While most existing TTA methods for video primarily utilize visual supervisory signals, they often overlook the potential contribution of inherent audio data. To address this gap, we propose a novel approach that incorporates audio information into video TTA. Our method capitalizes on the rich semantic content of audio to generate audio-assisted pseudo-labels, a new concept in the context of video TTA. Specifically, we propose an audio-to-video label mapping method by first employing pre-trained audio models to classify audio signals extracted from videos and then mapping the audio-based predictions to video label spaces through large language models, thereby establishing a connection between the audio categories and video labels. To effectively leverage the generated pseudo-labels, we present a flexible adaptation cycle that determines the optimal number of adaptation iterations for each sample, based on changes in loss and consistency across different views. This enables a customized adaptation process for each sample. Experimental results on two widely used datasets (UCF101-C and Kinetics-Sounds-C), as well as on two newly constructed audio-video TTA datasets (AVE-C and AVMIT-C) with various corruption types, demonstrate the superiority of our approach. Our method consistently improves adaptation performance across different video classification models and represents a significant step forward in integrating audio information into video TTA. The code and datasets will be made publicly available.

Most existing video moment retrieval (VMR) benchmark datasets face a common issue of sparse annotations-only a few moments being annotated. We argue that videos contain a broader range of meaningful moments that, if leveraged, could significantly enhance performance. Existing methods typically follow a generate-then-select paradigm, focusing primarily on generating moment-query pairs while neglecting the crucial aspect of selection. In this paper, we propose a new method, HyperAux, to yield auxiliary moment-query pairs by modeling the multi-modal hyper-interaction between video and language. Specifically, given a set of candidate moment-query pairs from a video, we construct a hypergraph with multiple hyperedges, each corresponding to a moment-query pair. Unlike traditional graphs where each edge connects only two nodes (frames or queries), each hyperedge connects multiple nodes, including all frames within a moment, semantically related frames outside the moment, and an input query. This design allows us to consider the frames within a moment as a whole, rather than modeling individual frame-query relationships separately. More importantly, constructing the relationships among all moment-query pairs within a video into a large hypergraph facilitates selecting higher-quality data from such pairs. On this hypergraph, we employ a hypergraph neural network to aggregate node information, update the hyperedge, and propagate video-language hyper-interactions to each connected node, resulting in context-aware node representations. This enables us to use node relevance to select high-quality moment-query pairs and refine the moments’ boundaries. We also exploit the discrepancy in semantic matching within and outside moments to construct a loss function for training the HGNN without human annotations. Our auxiliary data enhances the performance of twelve VMR models under fully-supervised, weakly-supervised, and zero-shot settings across three widely used VMR datasets: ActivityNet Captions, Charades-STA, and QVHighlights. We will release the source code and models publicly.

Facial expressions have been widely used for depression recognition because it is intuitive and convenient to access. Pupil diameter contains rich emotional information that is already reflected in facial video streams. However, the spatiotemporal correlation between pupillary changes and facial behavior changes induced by emotional stimuli has not been explored in existing studies. This paper presents a novel multimodal fusion algorithm - Trial Selection Tensor Canonical Correlation Analysis (TSTCCA) to optimize the feature space and build a more robust depression recognition model, which innovatively combines the spatiotemporal relevance and complementarity between facial expression and pupil diameter features. TSTCCA explores the interaction between trials and obtains an effective fusion representation of two modalities from a trial subset related to depression. The experimental results show that TSTCCA achieves the highest accuracy of 78.81% with the subset of 25 trials.

In mixed-autonomy traffic environments, accurately predicting the lane change behavior of human-driven vehicles is critical for ensuring the safety and reliability of autonomous vehicle decision-making. However, existing approaches face two major challenges: 1) they tend to represent the relationships between the target vehicle and surrounding vehicles using parameters like relative position and speed. This approach either requires a fixed number of surrounding vehicles or introduces significant noise by relying on virtual vehicles; and 2) they often fail to fully exploit the vast amount of available vehicle trajectory data, leaving the complexities of vehicular interactions underexplored. To address these issues, this paper presents a novel lane change prediction framework using Transformer-based transfer learning. Our design aims to leverage inter-vehicle interactions learned from trajectory data to improve lane-change prediction accuracy. Specifically, pre-trained trajectory prediction models are used to adapt dynamically to the varying number of surrounding vehicles and to capture interaction context from large sets of trajectory data. We then refine the Transformer model to integrate this context and predict the target vehicle’s lane change intentions. The Transformer encoder transforms trajectory interaction context into a lane-change-oriented context using aggregated multi-head attention. The Transformer decoder, in turn, utilizes this context alongside the target vehicle’s states through relation-aware multi-head attention to forecast lane change behavior. Extensive experiments on two real-world datasets demonstrate that our proposed framework outperforms state-of-the-art baselines in both accuracy and robustness.

Ensuring the smooth operation of road traffic is a momentous target in Intelligent Transportation Systems, which can be expedited by a secure and reliable Internet of Vehicles (IoV). As prominent carriers of the IoV, intelligent vehicles (IVs), that bear the promising potential for alleviating traffic congestion, have become the core road traffic participants. However, the mixed-traffic environment escalates the risk of IVs, as the discretionary lane change behaviors of nearby human-driven vehicles may result in collisions with IVs, compromising the robust performance of the IoV. Recent studies have utilized advanced deep learning techniques to achieve proactive lane change intention prediction, including Recurrent Neural Networks and Transformer. Although attaining reasonable prediction performance, they adopt the data-driven paradigm, which excessively focuses on learning from data while neglecting the domain knowledge. Against this background, we propose to employ the knowledge-driven paradigm and design KLEP, a knowledge-driven lane change prediction framework. KLEP incorporates driving knowledge into lane change modeling, presenting the top-down hierarchical cognitive process of drivers when performing lane change maneuvers. Extensive experiments conducted on two real-world natural driving datasets demonstrate the effectiveness of KLEP. Compared to state-of-the-art lane change prediction baselines, KLEP consistently outperforms them and achieves average improvements of 6.2-7.1% and 53.0-67.2% on intention classification and intention forecast tasks across different datasets, respectively. We also validate that KLEP has strong interpretability that aligns with real-world physical laws in lane change scenarios and is lightweight enough to fulfill online prediction.

Multimodal recommender systems utilize a variety of information types to model user preferences and item properties, aiding in the discovery of items that align with user interests. Rich multimodal information alleviates inherent challenges in recommendation systems, such as data sparsity and cold start problems. However, multimodal information further introduces challenges in terms of robustness and generalization capability. Regarding robustness, multimodal information magnifies the risks associated with information adjustment and inherent noise, posing severe challenges to the stability of recommendation models. For generalization capability, multimodal recommender systems are more complex and difficult to train, making it harder for models to handle data beyond the training set, posing significant challenges to model generalization capability. In this paper, we analyze the shortcomings of existing robustness and generalization capability enhancement strategies in the multimodal recommendation field. We propose a sharpness-aware minimization strategy focused on batch data (BSAM), which effectively enhances the robustness and generalization capability of multimodal recommender systems without requiring extensive hyper-parameter tuning. Furthermore, we introduce a mixed loss variant strategy (BSAM+), which accelerates convergence and achieves remarkable performance improvement. We provide rigorous theoretical proofs and conduct experiments with nine advanced models on five widely used datasets to validate the superiority of our strategies. Moreover, our strategies can be integrated with existing robust training and data augmentation strategies to achieve further improvement, providing a superior training paradigm for multimodal recommendations.

In the era of Big Data, large-scale machine learning models have revolutionized various fields, driving significant advancements. However, large-scale model training demands high financial and computational resources, which are only affordable by a few technological giants and well-funded institutions. In this case, common users like mobile users, the real creators of valuable data, are often excluded from fully benefiting due to the barriers, while the current methods for accessing largescale models either limit user ownership or lack sustainability. This growing gap highlights the urgent need for a collaborative model training approach, allowing common users to train and share models. However, existing collaborative model training paradigms, especially federated learning (FL), primarily focus on data privacy and group-based model aggregation. To this end, this paper intends to address this issue by proposing a novel training paradigm named decentralized relay learning (DeRelayL), a sustainable learning system where permissionless participants can contribute to model training in a relay-like manner and share the model. In detail, this paper presents the architecture and workflow of DeRelayL, designs incentive mechanisms to ensure sustainability, and conducts theoretical analysis and numerical simulations to demonstrate its effectiveness

Recent advancements in Large Language Models (LLMs) have showcased their remarkable capabilities in text understanding and generation.However, even stronger LLMs are susceptible to acquiring erroneous or obsolete information from the training corpus.Direct secondary fine-tuning with data containing new knowledge may be ineffective in updating knowledge due to the conflict between old and new knowledge.In this paper, we propose a new paradigm for fine-tuning called F-Learning (Forgetting before Learning), which employs parametric arithmetic to facilitate the forgetting of old knowledge and learning of new knowledge.Experimental results on two publicly available datasets demonstrate that our proposed F-Learning can obviously improve the knowledge updating performance of both full fine-tuning and LoRA fine-tuning, simultaneously outperforming the existing baselines in most cases.Moreover, we have also discovered that forgetting old knowledge by subtracting the parameters of LoRA can yield a similar effect to subtracting the parameters of full fine-tuning, and occasionally even surpass it significantly.

Empathetic response generation is designed to comprehend the emotions of others and select the most appropriate strategies to assist them in resolving emotional challenges. Empathy can be categorized into cognitive empathy and affective empathy. The former pertains to the ability to understand and discern the emotional issues and situations of others, while the latter involves the capacity to provide comfort. To enhance one's empathetic abilities, it is essential to develop both these aspects. Therefore, we develop an innovative framework that combines retrieval augmentation and emotional support strategy integration. Our framework starts with the introduction of a comprehensive emotional palette for empathy. We then apply appraisal theory to decompose this palette and create a database of empathetic responses. This database serves as an external resource and enhances the LLM's empathy by integrating semantic retrieval mechanisms. Moreover, our framework places a strong emphasis on the proper articulation of response strategies. By incorporating emotional support strategies, we aim to enrich the model's capabilities in both cognitive and affective empathy, leading to a more nuanced and comprehensive empathetic response. Finally, we extract datasets ED and ET from the empathetic dialogue dataset EmpatheticDialogues and ExTES based on dialogue length. Experiments demonstrate that our framework can enhance the empathy ability of LLMs from both cognitive and affective empathy perspectives. Our code is released at https://github.com/CAS-SIAT-XinHai/APTNESS.

Reinforcement learning from human feedback (RLHF) is a crucial technique in aligning large language models (LLMs) with human preferences, ensuring these LLMs behave in beneficial and comprehensible ways to users. However, a longstanding challenge in human alignment techniques based on reinforcement learning lies in their inherent complexity and difficulty in training. To address this challenge, we present a simple yet effective Contrastive Learning Framework for Human Alignment (CLHA) to align LLMs with human preferences directly. CLHA employs a novel rescoring strategy to evaluate the noise within the data by considering its inherent quality and dynamically adjusting the training process. Simultaneously, CLHA utilizes pairwise contrastive loss and adaptive supervised fine-tuning loss to adaptively modify the likelihood of generating responses, ensuring enhanced alignment with human preferences. Using advanced methods, CLHA surpasses other algorithms, showcasing superior performance in terms of reward model scores, automatic evaluations, and human assessments on the widely used “Helpful and Harmless” dataset.

In this paper, we introduce an innovative pre-training framework TP-Link, which aims to improve context-dependent Text-to-SQL Parsing by leveraging Linking information. This enhancement is achieved through better representation of both natural language utterances and the database schema, ultimately facilitating more effective text-to-SQL conversations. We present two novel pre-training objectives: (i) utterance linking prediction (ULP) task that models intricate syntactic relationships among natural language utterances in context-dependent text-to-SQL scenarios, and (ii) schema linking prediction (SLP) task that focuses on capturing fine-grained schema linking relationships between the utterances and the database schema. Extensive experiments demonstrate that our proposed TP-Link achieves state-of-the-art performance on two leading downstream benchmarks (i.e., SParC and CoSQL).

Large language models (LLMs) have demonstrated impressive performance in various natural language processing (NLP) tasks. However, there is limited understanding of how well LLMs perform in specific domains (e.g, the intellectual property (IP) domain). In this paper, we contribute a new benchmark, the first Multilingual-oriented quiZ on Intellectual Property (MoZIP), for the evaluation of LLMs in the IP domain. The MoZIP benchmark includes three challenging tasks: IP multiple-choice quiz (IPQuiz), IP question answering (IPQA), and patent matching (PatentMatch). In addition, we also develop a new IP-oriented multilingual large language model (called MoZi), which is a BLOOMZ-based model that has been supervised fine-tuned with multilingual IP-related text data. We evaluate our proposed MoZi model and four well-known LLMs (i.e., BLOOMZ, BELLE, ChatGLM and ChatGPT) on the MoZIP benchmark. Experimental results demonstrate that MoZi outperforms BLOOMZ, BELLE and ChatGLM by a noticeable margin, while it had lower scores compared with ChatGPT. Notably, the performance of current LLMs on the MoZIP benchmark has much room for improvement, and even the most powerful ChatGPT does not reach the passing level. Our source code, data, and models are available at https://github.com/AI-for-Science/MoZi.

Among the various pre-trained neural language models that are popular today, dropout is already an indispensable regularization technique. To solve the inconsistency between training and inference caused by the randomness of dropout, some studies use consistency training to regularize dropout at the output layer. In this paper, we propose a novel Layer-wise Regularized Dropout (LR-Drop), which is specially designed for Transformer-based Language models. Specifically, LR-Drop layer-wise regularizes each Transformer layer using the consistency training strategy. Each training sample passes through the two siamese sub-models sampled by dropout, and then LR-Drop forces the hidden states, multi-head attention matrices, and output distribution of the two siamese sub-models to be consistent. The proposed LR-Drop can be regarded as a “self-distillation” framework, in which each sub-model generated by dropout is the other’s “teacher” model and “student” model. Through extensive experiments on 8 natural language understanding datasets, 6 neural machine translation datasets, and 1 abstractive summarization dataset (a total of 15 datasets), we show that LR-Drop achieves superior performances, including state-of-the-art results.

Learning behavior in legged robots presents a significant challenge due to its inherent instability and complex constraints. Recent research has proposed the use of a large language model (LLM) to generate reward functions in reinforcement learning, thereby replacing the need for manually designed rewards by experts. However, this approach, which relies on textual descriptions to define learning objectives, fails to achieve controllable and precise behavior learning with clear directionality. In this paper, we introduce a new video2reward method, which directly generates reward functions from videos depicting the behaviors to be mimicked and learned. Specifically, we first process videos containing the target behaviors, converting the motion information of individuals in the videos into keypoint trajectories represented as coordinates through a video2text transforming module. These trajectories are then fed into an LLM to generate the reward function, which in turn is used to train the policy. To enhance the quality of the reward function, we develop a video-assisted iterative reward refinement scheme that visually assesses the learned behaviors and provides textual feedback to the LLM. This feedback guides the LLM to continually refine the reward function, ultimately facilitating more efficient behavior learning. Experimental results on tasks involving bipedal and quadrupedal robot motion control demonstrate that our method surpasses the performance of state-of-the-art LLM-based reward generation methods by over 37.6% in terms of human normalized score. More importantly, by switching video inputs, we find our method can rapidly learn diverse motion behaviors such as walking and running.

Online chatting has become an essential aspect of our daily interactions, with stickers emerging as a prevalent tool for conveying emotions more vividly than plain text. While conventional image emotion recognition focuses on global features, sticker emotion recognition necessitates incorporating both global and local features, along with additional modalities like text. To address this, we introduce a topic ID-guided transformer method to facilitate a more nuanced analysis of the stickers. Since each sticker will have a topic, and the same topic will have the same object, we introduce a topic ID as a flag to group images by theme. Our approach encompasses a novel topic-guided context-aware module and a topic-guided attention mechanism, enabling the extraction of comprehensive topic context features from stickers sharing the same topic ID, significantly enhancing emotion recognition accuracy. Moreover, we integrate a frequency linear attention module to leverage frequency domain information to capture better the object information of the stickers and a locally enhanced re-attention mechanism for improved local feature extraction. Extensive experiments and ablation studies on the large-scale sticker emotion dataset SER30k validate the efficacy of our method. Experimental results show that our proposed method obtains the best accuracy on both single-modal and multi-modal sticker emotion recognition.

The rapid advancements in the development of multimodal large language models (MLLMs) have consistently led to new breakthroughs on various benchmarks. In response, numerous challenging and comprehensive benchmarks have been proposed to more accurately assess the capabilities of MLLMs. However, there is a dearth of exploration of the higher-order perceptual capabilities of MLLMs. To fill this gap, we propose the Image Implication understanding Benchmark, II-Bench, which aims to evaluate the model's higher-order perception of images. Through extensive experiments on II-Bench across multiple MLLMs, we have made significant findings. Initially, a substantial gap is observed between the performance of MLLMs and humans on II-Bench. The pinnacle accuracy of MLLMs attains 74.8%, whereas human accuracy averages 90%, peaking at an impressive 98%. Subsequently, MLLMs perform worse on abstract and complex images, suggesting limitations in their ability to understand high-level semantics and capture image details. Finally, it is observed that most models exhibit enhanced accuracy when image sentiment polarity hints are incorporated into the prompts. This observation underscores a notable deficiency in their inherent understanding of image sentiment. We believe that II-Bench will inspire the community to develop the next generation of MLLMs, advancing the journey towards expert artificial general intelligence (AGI). II-Bench is publicly available at https://huggingface.co/datasets/m-a-p/II-Bench.

Smartphones are widely used as portable data collectors for wearable and healthcare sensors that can passively collect data streams related to the environment, health status, and behaviors. Recent research shows that the collected data can be used to monitor not only the physical states but also the mental health of individuals. However, extracting the features of digital phenotypes that characterize major depressive disorder (MDD) is technically challenging and may raise significant privacy concerns. Addressing such challenges has become the focus of many researchers. This article provides a comprehensive analysis of several key issues related to ubiquitous sensing to aid in detecting MDD. Specifically, this article analyzes existing methodologies and feature extraction algorithms used to detect possible MDD through digital phenotyping from smartphone data. In particular, five types of features are summarized and explained, namely, location, movement, rhythm, sleep, and social and device usage. Finally, related limitations and challenges are discussed to provide paths for further research and engineering.

Recently, depression recognition through EEG has gained significant attention. However, two challenges have not been properly addressed in prior automated depression recognition and classification studies: (1) EEG data lacks an explicit topological structure. (2) Capturing spatio-temporal features of EEG signals is difficult. In this paper, we propose Multi-scale Adaptive Spatial-Temporal Graph Convolutional Network (MAST-GCN) for mining latent topological structure among EEG channels and capturing discriminative spatio-temporal features. First, we integrate Adaptive Graph Convolution (AGC) that merges the inherent graph construction method with a data-driven graph reconstruction method. The model uses attention mechanism to learn an adaptive topological structure and semantic information from different layers and classes. Second, we propose Multi-Scale Time Convolutional Layer (MS-TCL), which captures long-term dependence from EEG data. Since Graph Convolution is weak for aggregating the spatio-temporal information, we have implemented a 3D Graph Convolution (G3D) to directly capture the spatio-temporal dependencies by reconstructing the spatio-temporal graph. The experimental results demonstrate that MAST-GCN consistently outperforms state-of-the-art methods on two datasets. Furthermore, we use the gradient-based saliency maps for interpretability analysis, discovering the active brain regions and important electrode pairs related to depression.

Human gait refers to a daily motion that represents not only mobility but can also be used to identify the walker by either human observers or computers. Recent studies reveal that gait even conveys information about the walker’s emotion. Individuals in different emotion states may show different gait patterns. The mapping between various emotions and gait patterns provides a new source for automated emotion recognition. Compared to traditional emotion detection biometrics, such as facial expression, speech, and physiological parameters, gait is remotely observable, more difficult to imitate, and requires less cooperation from the subject. These advantages make gait a promising source for emotion detection. This article reviews current research on gait-based emotion detection, particularly on how gait parameters can be affected by different emotion states and how the emotion states can be recognized through distinct gait patterns. We focus on the detailed methods and techniques applied in the whole process of emotion recognition: data collection, preprocessing, and classification. Finally, we discuss possible future developments of efficient and effective gait-based emotion recognition using state-of-the-art techniques in intelligent computation and big data.

The human gait reflects substantial information about individual emotions. Current gait emotion recognition methods focus on capturing gait topology information and ignore the importance of fine-grained temporal features. This article proposes the temporal-tightly graph convolutional network (TT-GCN) to extract temporal features. TT-GCN comprises three significant mechanisms: the causal temporal convolution network (casual-TCN), the walking direction recognition auxiliary task, and the feature mapping layer. To obtain tight temporal dependencies and enhance the relevance among gait periods, the causal-TCN is introduced. Based on the assumption of emotional consistency in the walking directions, the auxiliary task is proposed to enhance the ability of fine-grained feature extraction. Through the feature mapping layer, affective features can be mapped into the appropriate representation and fused with deep learning features. TT-GCN shows the best performance across five comprehensive metrics. All experimental results verify the necessity and feasibility of exploring fine-grained temporal feature extraction.

Depression is one of the most common mental disorders, with sleep disturbances as typical symptoms. With the popularity of wearable devices increasing in recent years, more and more people wear portable devices to track sleep quality. Based on this, we believe that depression detection through wearable sleep data is more intelligent and economical. However, the majority of wearable devices face the problem of missing data during the data collection process. Otherwise, most existing studies of depression identification focus on the utilization of complex data, making it difficult to generalize and susceptible to noise interference. To address these issues, we propose a systematic ensemble classification model for depression (ECD). For the missing data problem of wearable devices, we design an improved GAIN method to further control the generation range of interpolated values, which can achieve a more reasonable treatment of missing values. Compared with the original GAIN approach, the improved method shows a 28.56% improvement when using MAE as the metric. For depression recognition, we use ensemble learning to construct a depression classification model which combines five classification models, including SVM, KNN, LR, CBR, and DT. Ensemble learning can improve the model's robustness and generalization. The voting mechanism is used in several places to improve noise immunity. The final classification model performed great on the dataset, with a precision of 92.55% and a recall of 91.89%. These results illustrate how efficient this method is in automatically detecting depression.

Recently, psychophysiological computing has received considerable attention. Due to easy acquisition at a distance and less conscious initiation, gait-based emotion recognition is considered as a valuable research branch in the field of psychophysiological computing. However, most existing methods rarely explore the spatio-temporal context of gait, which limits the ability to capture the higher-order relationship between emotion and gait. In this paper, we utilize a range of research, including psychophysiological computing and artificial intelligence, to propose an integrated emotion perception framework called EPIC, which can find novel joint topology and generate thousands of synthetic gaits by spatio-temporal interaction context. First, we analyze the joint coupling among non-adjacent joints by calculating Phase Lag Index (PLI), which can discover the latent connection among body joints. Second, to synthesize more sophisticated and accurate gait sequences, we explore the effect of spatio-temporal constraints, and propose a new loss function that utilizes the Dynamic Time Warping (DTW) algorithm and pseudo-velocity curve to constrain the output of Gated Recurrent Units (GRU). Finally, Spatial Temporal Graph Convolution Networks (ST-GCN) is used to classify emotions using the generation and the real data. Experimental results demonstrate our approach achieves the accuracy of 89.66%, and outperforms the state-of-the-art methods on Emotion-Gait dataset.

The rapid spread of the COVID-19 has not only affected personal health and economy, but also revolutionized people’s lifestyles. As more people turn to work and socialize online, the development of unmanned technologies based on the Internet of Vehicles (IoV), such as unmanned delivery, unmanned vehicles, unmanned transportation, etc., will become an inevitable trend. However, all kinds of intelligent terminals for unmanned equipment require a large amount of data interaction with devices such as cloud servers, mobile terminals, and roadside terminals, which poses cyber security risks. Furthermore, the outbreak of COVID-19 has prompted people to put forward higher demands for the security of network communications. Unfortunately, the current intrusion detection methods based on machine learning still have weaknesses such as low accuracy and low efficiency when faced with unbalanced data distribution. To solve the above problems, we propose a novel Tree-based BLS (TBLS) intrusion detection method according to the idea of ensemble learning and decision tree (CART and J48). The performance of TBLS was tested on the NSL-KDD dataset and the UNSW-NB15 dataset respectively, which contain a variety of malicious traffic types for attacks on the IoV. The results show that our proposed method can achieve higher accuracy rate and lower false alarm rate, compared with the existing 16 solutions.

A wireless federated learning system is investigated by allowing a server and multiple workers to exchange uncoded information via orthogonal wireless channels. Since the workers frequently upload local gradients to the server via band-limited channels, the uplink transmission from the workers to the server becomes a communication bottleneck. Therefore, a one-shot distributed principle component analysis (PCA) is leveraged to reduce the dimension of uploaded gradients to relieve the communication bottleneck. A PCA-based wireless federated learning (PCA-WFL) algorithm and its accelerated version (i.e., PCA-AWFL) are proposed based on the low-dimensional gradients and the Nesterov’s momentum. For the non-convex empirical risk, a finite-time analysis is performed to quantify the impacts of system hyper-parameters on the convergence of the PCA-WFL and PCA-AWFL algorithms. The PCA-AWFL algorithm is theoretically certified to converge faster than the PCA-WFL algorithm. Besides, the convergence rates of PCA-WFL and PCA-AWFL algorithms quantitatively reveal the linear speedup with respect to the number of workers over the vanilla gradient descent algorithm. Numerical results are used to demonstrate the improved convergence rates of the proposed PCA-WFL and PCA-AWFL algorithms over the benchmarks.

This paper focuses on contrastive learning for gait-based emotion recognition. The existing contrastive learning approaches are rarely suitable for learning skeleton-based gait representations, which suffer from limited gait diversity and inconsistent semantics. In this paper, we propose a Cross-coordinate contrastive learning framework utilizing Ambiguity samples for self-supervised Gait-based Emotion representation (CAGE). First, we propose ambiguity transform to push positive samples into ambiguous semantic space. By learning similarities between ambiguity samples and positive samples, our model can learn higher-level semantics of the gait sequences and maintain semantic diversity. Second, to encourage learning the semantic invariance, we uniquely propose cross-coordinate contrastive learning between the Cartesian coordinate and the Spherical coordinate, which brings rich supervisory signals to learn the intrinsic semantic consistency information. Exhaustive experiments show that CAGE improves existing self-supervised methods by 5%–10% accuracy, and it achieves comparable or even superior performance to supervised methods.

The accuracy and availability are the most critical and challenging problems for major depressive disorder (MDD) diagnosis. Limited receptive field and inaccurate visual interpretation always weaken the clinical application of deep learning-based depression recognition model. Thus, we propose a visually interpretable depression monitoring model termed Transformer and Convolutional with slot-attention (TC-slot) to assess depression from facial images. Specifically, this approach stands upon the intersection of convolution and transformer, combines self-attention mechanism and deep convolution, and uses a well-designed stem structure to explore the global and local relationships. Moreover, in TC-slot, a classifier built on slot-attention mechanism directly involved in the decision-making process further localizes salient regions of facial depression patterns and provides precise and meaningful explanations. The results indicate that the proposed approach effectively improves the classification and recognition performance compared with other state-of-the-art approaches, with guaranteed favorable visual interpretability, providing clinical insights into the assessment of the assessing depression.

Action recognition research has gained significant attention with two dominant unimodal approaches: skeleton-based and RGB video-based. While the former is known for its robustness in complex backgrounds, the latter provides rich environmental information useful for context-based analysis. However, the fusion of these two modalities remains an open challenge. In this paper, we propose a Spatial Transformer & Selective Temporal encoder (ST&ST) for skeleton-based action recognition by constructing two modules: Reranking-Enhanced Dynamic Mask Transformer (RE-DMT) and Selective Kernel Temporal Convolution (SK-TC). The RE-DMT captures global spatial features, while the dynamic mask strategy and reranking strategy reduce redundancy. The SK-TC captures both long-term and short-term temporal features and enables adaptive fusion. Furthermore, in two phases, we propose a Homogeneous-Heterogeneous Multimodal Network (HHMNet) for multi-modal action recognition. In the first phase, contrastive learning is employed to achieve implicit semantic fusion within the four homogeneous skeletal modalities (joint, bone, etc.). In the second phase, the fusion of heterogeneous modalities (skeleton & RGB video) is carried out at three levels: model, feature, and decision. At the model level, the powerful skeleton-based model from the previous phase provides explicit attention guidance to the RGB video-based model. At the feature level, multi-part contrastive learning enables semantic distillation between heterogeneous modalities. At the decision level, ensemble learning combines outputs for final action recognition. We evaluate our proposed ST&ST guided HHMNet on NTU RGB+D 60 & 120 and NW-UCLA datasets and demonstrate that it achieves state-of-the-art performance in both skeleton-based and multi-modal action recognition tasks.

Credit transactions are vital financial activities that yield substantial economic benefits. To further improve lending decisions, stakeholders require accurate and interpretable credit scoring methods. While the majority of previous studies have focused on the relationship between individual features and credit risk, only a few have investigated cross-features. Notably, cross-features can not only represent structured data effectively but also provide richer semantic information than individual features. Nevertheless, most previous methods for learning cross-feature effects from credit data have been implicit and unexplainable. This paper proposes a new credit scoring model based on contrastive augmentation and tree-enhanced embedding mechanisms, termed CATE. The proposed model automatically constructs explainable cross-features by using tree-based models to learn decision rules from the data. Moreover, the importance of each local cross-feature is then derived through an attention mechanism . Finally, the credit score of a user is evaluated using embedding vectors. Experimental results on 4 public datasets demonstrated the interpretability of our proposed method and outperformed 13 state-of-the-art benchmark methods in terms of performance.

The dramatic progresses of wireless technologies and wearable devices have significantly promoted the development and popularity of smart home, while digital twin (DT) emerges as a game changer benefiting from its enhanced capabilities of visualization and interaction. The DT is able to build a realtime and continuous visual replica of a physical object or process, and to provide realtime monitoring, anomaly prediction, smart interaction, and lifecycle management. This paper presents a DT model to empower healthcare monitoring in the smart home with the goals of graphical monitoring, healthcare prediction, and intelligent control. High fidelity DT of the house and its equipments is created for visualized monitoring, and two suites of devices are deployed for continuously acquiring the users’ electrocardiograph (ECG) waves and the WiFi signals in the house. Two intelligent algorithms are then developed to perform fall detection from WiFi signals and to screen atrial fibrillation from ECG waves collected by wearable devices. Experimental results well validate the proposed model’s effectiveness for smart home monitoring, and the advantages of the developed smart algorithms for healthcare prediction over counterparts.

Effective estimation of brain network connectivity enables better unraveling of the extraordinary complexity interactions of brain regions and helps in auxiliary diagnosis of psychiatric disorders. Considering different modalities can provide comprehensive characterizations of brain connectivity, we propose the message-passing-based nonlinear network fusion (MP-NNF) algorithm to estimate multimodal brain network connectivity. In the proposed method, the initial functional and structural networks were computed from fMRI and DTI separately. Then, we update every unimodal network iteratively, making it more similar to the others in every iteration, and finally converge to one unified network. The estimated brain connectivities integrate complementary information from multiple modalities while preserving their original structure, by adding the strong connectivities present in unimodal brain networks and eliminating the weak connectivities. The effectiveness of the method was evaluated by applying the learned brain connectivity for the classification of major depressive disorder (MDD). Specifically, 82.18% classification accuracy was achieved even with the simple feature selection and classification pipeline, which significantly outperforms the competing methods. Exploration of brain connectivity contributed to MDD identification suggests that the proposed method not only improves the classification performance but also was sensitive to critical disease-related neuroimaging biomarkers.

Accurate short-term traffic flow prediction plays an important role in providing road condition information in the immediate future. With the information, intelligent vehicles can plan and adjust the route to prevent congestion. As a result, many models for short-term traffic flow forecasting have been proposed to date. However, most of them focus on the prediction of the entire traffic network, which could lead to several problems: (1) the entire traffic network could have a large scale and a complex structure, for which the model training is likely to be time-consuming as well as inefficient; (2) processing a large amount of training data on the central cloud could cause much calculation pressure on the server and increase the risk of privacy leakage. In this paper, we propose a Spatial-Temporal Attention Graph Convolution Network on Edge Cloud model (STAGCN-EC). We first divide the entire traffic network into several parts to reduce its scale and complexity. Then, we allocate each part of the network to a certain Roadside Unit (RSU) for training, thus there is no need to process all data on the central server. Besides, we utilize spatial-temporal attention and features extracting module that fits the low computational power devices like RSUs, to capture spatial-temporal dependence and predict traffic flow. At last, we use two highway datasets from District 7 and District 4 in California to validate our model. Through the experiments, we find out that our model performs well both in predicted precision and efficiency compared with the five baseline methods.

The accurate trajectory prediction of surrounding vehicles is crucial for the sustainability and safety of connected and autonomous vehicles under mixed traffic streams in the real world. The task of trajectory prediction is challenging because there are all kinds of factors affecting the motions of vehicles, such as the individual movements, the ambient driving environment especially road conditions, and the interactions with neighboring vehicles. To resolve the above issues, this work proposes a novel Heterogeneous Context-Aware Graph Convolutional Networks following the Encoder-Decoder architecture, which simultaneously extracts the hidden contexts from individual historical trajectories, varying driving scene, and inter-vehicle interactional behaviors. Specifically, the historical vehicle trajectories are fed into Temporal Convolutional Network to capture the individual context. Besides, a 2-Dimensional Convolutional Network with temporal attention is designed for transforming the scene image stream into compressing scene context. Then a Spatio-Temporal Dynamic Graph Convolutional Networks is devised to model the evolving interactional patterns, which incorporates the acquired individual and scene contexts as the representation of the node. Finally, the aforementioned three contexts are combined and fed into the decoder to produce future trajectories. The proposed model is validated on two real-world datasets which contain various driving scenarios. Results demonstrated that the proposed model outperforms state-of-the-art methods in prediction accuracy and achieves immense stability towards different vehicle states.

We focus on unsupervised representation learning for skeleton based action recognition. Existing unsupervised approaches usually learn action representations by motion prediction but they lack the ability to fully learn inherent semantic similarity. In this paper, we propose a novel framework named Prototypical Contrast and Reverse Prediction (PCRP) to address this challenge. Different from plain motion prediction, PCRP performs reverse motion prediction based on encoder-decoder structure to extract more discriminative temporal pattern, and derives action prototypes by clustering to explore the inherent action similarity within the action encoding. Specifically, we regard action prototypes as latent variables and formulate PCRP as an expectation-maximization (EM) task. PCRP iteratively runs (1) E-step as to determine the distribution of action prototypes by clustering action encoding from the encoder while estimating concentration around prototypes, and (2) M-step as optimizing the model by minimizing the proposed ProtoMAE loss, which helps simultaneously pull the action encoding closer to its assigned prototype by contrastive learning and perform reverse motion prediction task. Besides, the sorting can also serve as a temporal task similar as reverse prediction in the proposed framework. Extensive experiments on N-UCLA, NTU 60, and NTU 120 dataset present that PCRP outperforms main stream unsupervised methods and even achieves superior performance over many supervised methods. The codes are available at:https://github.com/LZUSIAT/PCRP.

Medical image segmentation methods downsample images for feature extraction and then upsample them to restore resolution for pixel-level predictions. In such schema, upsample technique is vital in restoring information for better performance. However, existing upsample techniques leverage little information from downsampling paths. The local and detailed feature from the shallower layer such as boundary and tissue texture is crucial in segmentation, especially medical image segmentation. To this end, we propose a novel upsample approach for medical image segmentation, Window Attention Upsample (WAU), which upsamples features conditioned on local and detailed features from downsampling path in local windows by introducing attention decoders of Transformer. WAU could serve as a general upsample method and be incorporated into any segmentation model that possesses lateral connections. We first propose the Attention Upsample which consists of Attention Decoder (AD) and bilinear upsample. AD leverages pixel-level attention to model longrange dependency and global information for a better upsample. Bilinear upsample is introduced as the residual connection to complement the upsampled features. Moreover, considering the extensive memory and computation cost of pixel-level attention, we further design a window attention scheme to restrict attention computation in local windows instead of the global range. We evaluate our method (WAU) on classic UNet structure with lateral connections and achieve state-of-the-art performance on Medical Segmentation Decathlon (MSD) Brain and Automatic Cardiac Diagnosis Challenge (ACDC) datasets. We also validate the effectiveness of our method on multiple classic architectures and achieve consistent improvement.

Math Word Problems (MWP) aims to automatically solve mathematical questions given in texts. Previous studies tend to design complex models to capture additional information in the original text so as to enable the model to gain more comprehensive features. In this paper, we turn our attention in the opposite direction, and work on how to discard redundant features containing spurious correlations for MWP. To this end, we design an Expression Syntax Information Bottleneck method for MWP (called ESIB) based on variational information bottleneck, which extracts essential features of the expression syntax tree while filtering latent-specific redundancy containing syntax-irrelevant features. The key idea of ESIB is to encourage multiple models to predict the same expression syntax tree for different problem representations of the same problem by mutual learning so as to capture consistent information of expression syntax tree and discard latent-specific redundancy. To improve the generalization ability of the model and generate more diverse expressions, we design a self-distillation loss to encourage the model to rely more on the expression syntax information in the latent space. Experimental results on two large-scale benchmarks show that our model not only achieves state-of-the-art results but also generates more diverse solutions.

Person re-identification (Re-ID) via gait features within 3D skeleton sequences is a newly-emerging topic with several advantages. Existing solutions either rely on hand-crafted descriptors or supervised gait representation learning. This paper proposes a self-supervised gait encoding approach that can leverage unlabeled skeleton data to learn gait representations for person Re-ID. Specifically, we first create self-supervision by learning to reconstruct unlabeled skeleton sequences reversely, which involves richer high-level semantics to obtain better gait representations. Other pretext tasks are also explored to further improve self-supervised learning. Second, inspired by the fact that motion's continuity endows adjacent skeletons in one skeleton sequence and temporally consecutive skeleton sequences with higher correlations (referred as locality in 3D skeleton data), we propose a locality-aware attention mechanism and a locality-aware contrastive learning scheme, which aim to preserve locality-awareness on intra-sequence level and inter-sequence level respectively during self-supervised learning. Last, with context vectors learned by our locality-aware attention mechanism and contrastive learning scheme, a novel feature named Constrastive Attention-based Gait Encodings (CAGEs) is designed to represent gait effectively. Empirical evaluations show that our approach significantly outperforms skeleton-based counterparts by 15-40 percent Rank-1 accuracy, and it even achieves superior performance to numerous multi-modal methods with extra RGB or depth information. Our codes are available at https://github.com/Kali-Hac/Locality-Awareness-SGE.

Visual relationship detection (VRD) is one newly developed computer vision task, aiming to recognize relations or interactions between objects in an image. It is a further learning task after object recognition, and is important for fully understanding images even the visual world. It has numerous applications, such as image retrieval, machine vision in robotics, visual question answer (VQA), and visual reasoning. However, this problem is difficult since relationships are not definite, and the number of possible relations is much larger than objects. So the complete annotation for visual relationships is much more difficult, making this task hard to learn. Many approaches have been proposed to tackle this problem especially with the development of deep neural networks in recent years. In this survey, we first introduce the background of visual relations. Then, we present categorization and frameworks of deep learning models for visual relationship detection. The high-level applications, benchmark datasets, as well as empirical analysis are also introduced for comprehensive understanding of this task.

Currently, depression has become a common mental disorder, especially among postgraduates. It is reported that postgraduates have a higher risk of depression than the general public, and they are more sensitive to contact with others. Thus, a non-contact and effective method for detecting people at risk of depression becomes an urgent demand. In order to make the recognition of depression more reliable and convenient, we propose a multi-modal gait analysis-based depression detection method that combines skeleton modality and silhouette modality. Firstly, we propose a skeleton feature set to describe depression and train a Long Short-Term Memory (LSTM) model to conduct sequence strategy. Secondly, we generate Gait Energy Image (GEI) as silhouette features from RGB videos, and design two Convolutional Neural Network (CNN) models with a new loss function to extract silhouette features from front and side perspectives. Then, we construct a multi-modal fusion model consisting of fusing silhouettes from the front and side views at the feature level and the classification results of different modalities at the decision level. The proposed multi-modal model achieved accuracy at 85.45% in the dataset consisting of 200 postgraduate students (including 86 depressive ones), 5.17% higher than the best single-mode model. The multi-modal method also shows improved generalization by reducing the gender differences. Furthermore, we design a vivid 3D visualization of the gait skeletons, and our results imply that gait is a potent biometric for depression detection.

With the ability to acquire and process large-scale traffic big data, a cooperative intelligent transport system can be realized. Identifying important nodes in a traffic network contributes to better traffic control, which plays a more critical role in improving the traffic efficiency of a cooperative intelligent transport system. However, existing traffic node importance evaluation methods rely on manually designed metrics such as betweenness, degree, which may lead to biased results. Meanwhile, the traditional method of iteratively deleting nodes is unsuitable for a large-scale traffic network. In this paper, we propose a novel traffic node importance evaluation method based on clustering in represented transportation network. Specifically, the proposed method first construct a length-weighted network based on the geographic road network. Then, it learns the low-dimensional embeddings of nodes employing network representation learning. Finally, it clusters each nodes with a machine learning method and identifies the critical nodes through vehicle flow of different nodes. Experimental results on the real-word dataset show that our proposed method has excellent performance compared to baseline methods.

It is critical to reduce the operating cost of shared buses for bus companies and improve the user experience of passengers. However, existing studies focus on either bus scheduling or route planning, which cannot accomplish the above mentioned goals concurrently. In this paper, we construct a joint bus scheduling and route planning framework to maximize the number of passengers, minimize the total length of routes and the number of required buses, as well as guarantee good user experience of passengers. First, we establish a system model based on a real-world scenario and formulate a multi-objective combinational optimization problem. Then, based on the extracted traffic topology of urban traffic networks and the generated candidate line set, we propose an offline algorithm to cope with the similar passenger flow distributions, e.g., morning or evening peak of every day. In order to cope with dynamic real-time passenger flows, an online algorithm is designed. Experiments are carried out based on real-word scenarios. The results show that the proposed algorithms can greatly reduce the operating cost of bus companies and guarantee good user experience based on real-world scheduling data in comparison with several existing methods.

A variety of novel mobile applications are developed to attract the interests of potential users in the emerging 5G-enabled vehicular networks. Although computation offloading and task scheduling have been widely investigated, it is rather challenging to decide the optimal offloading ratio and perform adaptive task scheduling in high-dynamic networks. Furthermore, the scheduling policy made by the network operator may be violated, since vehicular users are rational and selfish to maximize their own profits. By considering the incentive compatibility and individual rationality of vehicular users, we present POETS, an efficient partial computation offloading and adaptive task scheduling algorithm to maximize the overall system-wide profit. Specially, a two-sided matching algorithm is first proposed to derive the optimal transmission scheduling discipline. After that, the offloading ratio of vehicular users can be obtained through convex optimization, without any information of other users. Furthermore, a non-cooperative game is constructed to derive the payoff of vehicular users that can reach the equilibrium between users and the network operator. Theoretical analyses and performance evaluations based on real-world traces of taxies demonstrate the effectiveness of our proposed solution.

Intelligent Transportation System (ITS) is critical to cope with traffic events, e.g., traffic jams and accidents, and provide services for personal traveling. However, existing researches have not jointly considered the user data safety, utility and system latency comprehensively, to the best of our knowledge. Since both safe and efficient transmissions are significant for ITS, we construct a blockchain-enabled crowdsensing framework for distributed traffic management. First, we illustrate the system model and formulate a multi-objective optimization problem. Due to its complexity, we decompose it into two subproblems, and propose the corresponding schemes, i.e., a Deep Reinforcement Learning (DRL)-based algorithm and a DIstributed Alternating Direction mEthod of Multipliers (DIADEM) algorithm. Extensive experiments are carried out to evaluate the performance of our solutions, and experimental results demonstrate that the DRL-based algorithm can legitimately select active miners and transactions to make a satisfied trade-off between the blockchain safety and latency, and the DIADEM algorithm can effectively select task computation modes for vehicles in a distributed way to maximize their social welfare.

Skeleton-based person re-identification (Re-ID) is an emerging open topic providing great value for safety-critical applications. Existing methods typically extract hand-crafted features or model skeleton dynamics from the trajectory of body joints, while they rarely explore valuable relation information contained in body structure or motion. To fully explore body relations, we construct graphs to model human skeletons from different levels, and for the first time propose a Multi-level Graph encoding approach with Structural-Collaborative Relation learning (MG-SCR) to encode discriminative graph features for person Re-ID. Specifically, considering that structurally-connected body components are highly correlated in a skeleton, we first propose a multi-head structural relation layer to learn different relations of neighbor body-component nodes in graphs, which helps aggregate key correlative features for effective node representations. Second, inspired by the fact that body-component collaboration in walking usually carries recognizable patterns, we propose a cross-level collaborative relation layer to infer collaboration between different level components, so as to capture more discriminative skeleton graph features. Finally, to enhance graph dynamics encoding, we propose a novel self-supervised sparse sequential prediction task for model pre-training, which facilitates encoding high-level graph semantics for person Re-ID. MG-SCR outperforms state-of-the-art skeleton-based methods, and it achieves superior performance to many multi-modal methods that utilize extra RGB or depth features. Our codes are available at https://github.com/Kali-Hac/MG-SCR.

Person re-identification via 3D skeletons is an emerging topic with great potential in security-critical applications. Existing methods typically learn body and motion features from the body-joint trajectory, whereas they lack a systematic way to model body structure and underlying relations of body components beyond the scale of body joints. In this paper, we for the first time propose a Self-supervised Multi-scale Skeleton Graph Encoding (SM-SGE) framework that comprehensively models human body, component relations, and skeleton dynamics from unlabeled skeleton graphs of various scales to learn an effective skeleton representation for person Re-ID. Specifically, we first devise multi-scale skeleton graphs with coarse-to-fine human body partitions, which enables us to model body structure and skeleton dynamics at multiple levels. Second, to mine inherent correlations between body components in skeletal motion, we propose a multi-scale graph relation network to learn structural relations between adjacent body-component nodes and collaborative relations among nodes of different scales, so as to capture more discriminative skeleton graph features. Last, we propose a novel multi-scale skeleton reconstruction mechanism to enable our framework to encode skeleton dynamics and high-level semantics from unlabeled skeleton graphs, which encourages learning a discriminative skeleton representation for person Re-ID. Extensive experiments show that SM-SGE outperforms most state-of-the-art skeleton-based methods. We further demonstrate its effectiveness on 3D skeleton data estimated from large-scale RGB videos. Our codes are open at https://github.com/Kali-Hac/SM-SGE.

The rapid development of the COVID-19 pandemic has threatened the lives of people around the world. Many people were caught in anxiety and panic, which also prevents people from fully concentrating on their normal lives. However, the current common neurofeedback therapies used to solve the problem of lack of attention cannot fully deal with the differences in each individual. In addition, direct contact between the patient and the doctor also increases the risk of virus transmission during treatment. This article combines neurofeedback and IoT to establish an adaptive attention adjustment method. IoT connects patients and doctors remotely, reducing the direct contact between them. In order to adapt to individual differences, the feedback indicators of each individual are individually calibrated. In addition, the proportional, integral, and derivative controller was used to adjust the difficulty of the feedback task to adapt to each individual’s self-regulation ability and provide the individual with a higher level of regulation. We also designed adaptive attention adjustment experiments for different individuals. The results show that through adaptive feedback training, the individual’s feedback indicator has dropped by 77.90%, and the individual can adjust his attention state to the individual’s optimal baseline threshold, and the oscillation error gradually reduces to the expected threshold range. This method can cope with the differences between different individuals and provide each individual with the same level of feedback regulation. In the future, this study may provide a general adjuvant treatment for other mental illnesses.

The Internet of Medical Things (IoMT) is an extension and specialization of that original Internet of Things (IoT) concept, and applies to the interconnectedness of devices, software applications, and data which are specific to the medical industry. IoMT can add smart technologies to medical devices to monitor the progression of a disease away from the doctor’s office and learn things that could impact future care guidelines and patients. It can also provide a better way to care for our elderly by tracking vitals and heart performance, glucose and other body systems, and activity and sleeping levels. During the outbreak of pandemic (e.g., COVID-19), IoMT can even be used to detect main symptoms ubiquitously using intelligent sensors and trace the origin of the outbreak based on aggregated IoT data (e.g., geographic mobile data and purchase history). Although most of the contemporary IoMT systems can measure risks, make decisions, and take actions automatically, the lack of emotion-aware abilities will be an obstacle to more harmonious human–machine interaction and more efficient medical process. Besides, mental disorders, such as depression, schizophrenia, and anxiety, have become a more noticeable cause of suffering. The integration of emotion-aware abilities into IoMT can also contribute to monitor emotional dysregulation continuously in subjects with mental disorders or undergoing serious pandemic such as COVID-19, and give these patients personalized therapy recommendations. Research on affective computing has defined a framework to recognize, interpret, and process human affects, but more research is needed to investigate its application to biomedical applications, especially “in the wild” and over extended periods of time, and how to integrate emotion-aware abilities into IoMT organically is still an open question. This special issue aims to create a platform for researchers, developers, and practitioners from both academia and industry to disseminate the state-of-the-art results and to advance the Emotion-Aware ubiquitous computing in IoMT.

Action recognition is essential for many human-centered applications in the Internet of Things (IoT). Especially, in the Internet of Medical Things (IoMT), action recognition shows great importance in surgical assistance, patient monitoring, etc. Recently, 3-D skeleton sequence-based action recognition draws broad attention. It is a challenging task that needs effective modeling on intraframe skeleton representations and interframe temporal dynamics. Standard long short-term memory (LSTM)-based models are widely used for sequence modeling due to its long-term memory, yet they are unable to fully model the relationship between different body joints or persons to extract crucial co-occurrence features from different levels. To handle this shortcoming, we propose an attention-based multilevel co-occurrence graph convolutional LSTM (AMCGC-LSTM). By integrating graph convolutional networks (GCNs) into LSTM, the proposed model is capable of leveraging body structural information from skeletons and strengthening the multilevel co-occurrence (MC) feature learning. Specifically, we first design the spatial attention module for feature enhancement of key joints from skeleton inputs. Second, we design MC memory units coupled with GCN to automatically model the spatial relationship between joints, and simultaneously capture the co-occurrence features from different joints, persons, and frames. Finally, we construct aggregated features of MCs (AFMCs) from MC memory units to better represent the intraframe action context encoding, and leverage a concurrent LSTM (Co-LSTM) to further model their temporal dynamics for action recognition. Our model significantly outperforms mainstream methods on NTU RGB+D 60/120 data set, mutual action subset of NTU RGB+D 60/120 data set, and Northewestern-UCLA data set.

The Internet of Medical Things (IoMT) aims to exploit the Internet-of-Things (IoT) techniques to provide better medical treatment scheme for patients with smart, automatic, timely, and emotion-aware clinical services. One of the IoMT instances is applying IoT techniques to sleep-aware smartphones or wearable devices' applications to provide better sleep healthcare services. As we all know, sleep is vital to our daily health. What is more, studies have shown a strong relationship between sleep difficulties and various diseases such as COVID-19. Therefore, leveraging IoT techniques to develop a longer lifetime sleep healthcare IoMT system, with a tradeoff between data transferring/processing speed and battery energy efficiency, to provide longer time services for bad sleep condition persons, especially the COVID-19 patients or survivors, is a meaningful research topic. In this study, we propose an IoT-enabled sleep data fusion networks (SDFN) module with a star topology Bluetooth network to fuse data of sleep-aware applications. A machine learning model is built to detect sleep events through an audio signal. We design two data reprocessing mechanisms running on our IoT devices to alleviate the data jam problem and save the IoT devices' battery energy. The experiments manifest that the presented module and mechanisms can save the energy of the system and alleviate the data jam problem of the device.

The prompt evolution of Internet of Medical Things (IoMT) promotes pervasive in-home health monitoring networks. However, excessive requirements of patients result in insufficient spectrum resources and communication overload. Mobile Edge Computing (MEC) enabled 5G health monitoring is conceived as a favorable paradigm to tackle such an obstacle. In this paper, we construct a cost-efficient in-home health monitoring system for IoMT by dividing it into two sub-networks, i.e., intra-Wireless Body Area Networks (WBANs) and beyond-WBANs. Highlighting the characteristics of IoMT, the cost of patients depends on medical criticality, Age of Information (AoI) and energy consumption. For intra-WBANs, a cooperative game is formulated to allocate the wireless channel resources. While for beyond-WBANs, considering the individual rationality and potential selfishness, a decentralized non-cooperative game is proposed to minimize the system-wide cost in IoMT. We prove that the proposed algorithm can reach a Nash equilibrium. In addition, the upper bound of the algorithm time complexity and the number of patients benefiting from MEC is theoretically derived. Performance evaluations demonstrate the effectiveness of our proposed algorithm with respect to the system-wide cost and the number of patients benefiting from MEC.

Pervasive edge computing (PEC) is an emerging paradigm for the industrial Internet of Things (IIoT), and software-defined networks (SDN) offer lower latency services, and massive intelligent devices connectivity for the IIoT. However, the PEC has some issues with data security, and privacy while PEC devices sharing data among edges. What's more, the centralized SDN suffers from single point of attacks such as distributed denial of service (DDoS) from IIoT devices, and has the challenge of data leakage. In this article, we use blockchain, and proxy reencryption (PRE) technologies to tackle these challenges. The blockchain authorizes all devices in the network to improve their credibility, and authenticity. In addition, a blockchain-based data sharing framework that combines a PRE scheme is introduced for secure device-to-device communication in PEC environments. A series of smart contracts are designed for flexible operations of searching, and updating records on the blockchain. The experiments reveal that our design is highly efficient, and has high performance.

In this article, we investigate the jointly optimized resource allocation with hybrid multiple access in energy-efficient industrial Internet of Things (IIoT), where some devices (e.g., those for critical control devices) have higher transmission priority and stable energy supply while some devices (e.g., those for comprehensive sensors) may not. We consider a system model supporting wireless powered IIoT devices, with certain user terminal as a potential relay for the transmission between a hybrid access point and another user terminal. Constrained by the limited energy storage, the user needs to harvest energy before relaying and only the harvested energy is utilized for the following transmission. We propose a collaborative orthogonal and nonorthogonal multiple access protocol where two cooperation schemes with and without decoding the relay message are applied. Jointly considering time sharing in the transmission process, power splitting for simultaneous wireless information and power transfer, and transmit power allocation at the cooperative user, the achievable rate regions under the Rayleigh fading channel model are derived. Based on which, an optimization problem on resource allocation strategies is formulated and discussed. Both analytical and numerical results are provided, illustrating the impact of user geometry on the achievable rates as well as the optimal resource allocation with different cooperative strategies applied in different use cases. Aiming to enhance resource utilization, energy-efficient cooperation enables the combination of various transmission modes and networking classes in large scale networks, as well as a better use of ambient radio frequency signals for wireless powered transmissions.

Driver fatigue has become a leading cause of accidents and death in the transportation industry. Electroencephalography (EEG)-based fatigue detection can be a good way to reduce accidents and improve safety and efficiencies throughout the transportation system. In this article, we focus on investigating whether the spatial–temporal changes in the relations between EEG channels are specific to different driving states. EEG signals were first partitioned into several segments, and the covariance matrices obtained from each segment were input into a recurrent neural network to extract high-level temporal features. Meanwhile, the covariance matrices of whole signals were leveraged to extract spatial characteristics that were fused with temporal features to obtain comprehensive spatial–temporal information. In experiments on an open benchmark dataset, our method achieved an excellent classification accuracy of 89.28% and showed superior performance compared to several other state-of-the-art methods. These results indicate that our method can enable higher performance in driver fatigue detection.

Facial expression recognition has become a newly-emerging topic in recent decades, which has important value in the field of human-computer interaction. In this paper, we present a deep learning based approach, named frequency neural network (FreNet), for facial expression recognition. Different from convolutional neural network in spatial domain, FreNet inherits the advantages of processing image in frequency domain, such as efficient computation and spatial redundancy elimination. First, we propose the learnable multiplication kernel and construct multiple multiplication layers to learn features in frequency domain. Second, a summarization layer is proposed following multiplication layers to further yield high-level features. Third, based on the property of discrete cosine transform (DCT), we utilize multiplication layers and summarization layer to construct the Basic-FreNet, which can yield high-level features on the widely used DCT feature. Finally, to further achieve better performance on Basic-FreNet, we propose the Block-FreNet in which the weight-shared multiplication kernel is designed for feature learning and the block sub-sampling is designed for dimension reduction. The experimental results show that the Block-FreNet not only achieves superior performance, but also greatly reduces the computational cost. To our best knowledge, the proposed approach is the first attempt to fill in the blank of frequency based deep learning model for facial expression recognition.

Recently, Internet of Vehicles (IoV) has become one of the most active research fields in both academic and industry, which exploits resources of vehicles and Road Side Units (RSUs) to execute various vehicular applications. Due to the increasing number of vehicles and the asymmetrical distribution of traffic flows, it is essential for the network operator to design intelligent offloading strategies to improve network performance and provide high-quality services for users. However, the lack of global information and the time-variety of IoVs make it challenging to perform effective offloading and caching decisions under long-term energy constraints of RSUs. Since Artificial Intelligence (AI) and machine learning can greatly enhance the intelligence and the performance of IoVs, we push AI inspired computing, caching and communication resources to the proximity of smart vehicles, which jointly enable RSU peer offloading, vehicle-to-RSU offloading and content caching in the IoV framework. A Mix Integer Non-Linear Programming (MINLP) problem is formulated to minimize total network delay, consisting of communication delay, computation delay, network congestion delay and content downloading delay of all users. Then, we develop an online multi-decision making scheme (named OMEN) by leveraging Lyapunov optimization method to solve the formulated problem, and prove that OMEN achieves near-optimal performance. Leveraging strong cognition of AI, we put forward an imitation learning enabled branch-and-bound solution in edge intelligent IoVs to speed up the problem solving process with few training samples. Experimental results based on real-world traffic data demonstrate that our proposed method outperforms other methods from various aspects.

Recent developments of edge computing and content caching in wireless networks enable the Intelligent Transportation System (ITS) to provide high-quality services for vehicles. However, a variety of vehicular applications and time-varying network status make it challenging for ITS to allocate resources efficiently. Artificial intelligence algorithms, owning the cognitive capability for diverse and time-varying features of Internet of Connected Vehicles (IoCVs), enable an intent-based networking for ITS to tackle the above-mentioned challenges. In this paper, we develop an intent-based traffic control system by investigating Deep Reinforcement Learning (DRL) for 5G-envisioned IoCVs, which can dynamically orchestrate edge computing and content caching to improve the profits of Mobile Network Operator (MNO). By jointly analyzing MNO's revenue and users' quality of experience, we define a profit function to calculate the MNO's profits. After that, we formulate a joint optimization problem to maximize MNO's profits, and develop an intelligent traffic control scheme by investigating DRL, which can improve system profits of the MNO and allocate network resources effectively. Experimental results based on real traffic data demonstrate our designed system is efficient and well-performed.

The explosive growth of mobile devices promotes the prosperity of novel mobile applications, which can be realized by service offloading with the assistance of edge computing servers. However, due to limited computation and storage capabilities of a single server, long service latency hinders the continuous development of service offloading in mobile networks. By supporting multi-server cooperation, Pervasive Edge Computing (PEC) is promising to enable service migration in highly dynamic mobile networks. With the objective of maximizing the system utility, we formulate the optimization problem by jointly considering the constraints of server storage capability and service execution latency. To enable dynamic service placement, we first utilize Lyapunov optimization method to decompose the long-term optimization problem into a series of instant optimization problems. Then, a sample average approximation-based stochastic algorithm is proposed to approximate the future expected system utility. Afterwards, a distributed Markov approximation algorithm is utilized to determine the service placement configurations. Through theoretical analysis, the time complexity of our proposed algorithm is linear to the number of users, and the backlog queue of PEC servers is stable. Performance evaluations are conducted based on both synthetic and real trace-driven scenarios, with numerical results demonstrating the effectiveness of our proposed algorithm from various aspects.

Hypertension detection is an important global health challenge, in which there exists a large population of hypertensive people, yet a great portion of them remain undiagnosed. Recently, studies have indicated a strong correlation between early obstructive sleep apnea and the development of hypertension. Based on the relationship, we propose the smartphone hypertension detector which can detect hypertensive users by automatically monitoring user's overnight sleep behavior. The portable mobile application can serve as a premonitory detection method complementary to traditional medical detection methods. Experiments of smartphone hypertension detector have demonstrated its desired functionality and practicality for real-world deployment.

With the expansion of the IoT, it is important to optimize available bandwidth to reliably support edge to device communications. Thus we propose a wireless network where each edge server communicates with its end devices using its wireless band as a primary channel, assisted by a secondary edge server that can relay communications via its own wireless band as a secondary channel. The network can optimize capacity by balancing load between primary and secondary wireless bands, and we analyze the geometry of achievable rate regions, depending on the state of bands modeled as Rayleigh fading channels. The allocation of a connection to the primary or secondary band is formulated as an optimization problem which is then solved, and illustrated with numerical examples.

As the challenge of depression problems increases today, it is important to effective and timely detect for patients' treatments and depression prevention. Current methods of automated depression diagnosis depend almost entirely on audio, video, and Electroencephalogram (EEG) etc. In this paper, we propose a novel method to detect depression using gait data that is collected by Kinect camera. Its key components include a camera rectification method and a rigid-body representation of the human body. The camera rectification method achieves the purpose of improving data processing accuracy by changing the camera angle. The rigid-body representation can not only improve the robustness of detecting depression patients with noisy input, but also can reduce the classification time. We evaluate our method on the depression gait dataset in postgraduate students. The proposed method has an outstanding performance in classic machine learning algorithms, and the best accuracy can achieve 88.89%. Our solution provides a new method for automatic depression detection (ADD) that has exciting implications in clinical theory and practice, and has the advantages of high accuracy, inexpensive, low time cost, and no-contact.

Human action recognition plays an important role in e-health applications, such as surgical skill analysis, patient monitoring, and automatic nursing systems. Recently, skeleton-based action recognition gains massive attention. It is an essential yet challenging task that requires effectively modeling the intra-frame skeleton representation and inter-frame temporal dynamics. Traditional Long Short-Term Memory (LSTM) based methods mainly capture long-term action context information from global level, yet they cannot fully model the relationship between different joints or persons to mine crucial co-occurrence features from different levels. To overcome this drawback, we propose a general end-to-end Multi-level Co-occurrence Graph Convolutional LSTM (MCGC-LSTM). By incorporating graph convolutional networks (GCN) into LSTM, our model can not only better exploit body structural information from skeletons but also enhance the multi-level co-occurrence feature learning. Specifically, we first devise multi-level co-occurrence (MC) memory units coupled with GCN to automatically model the spatial relationship between joints, and simultaneously capture the co-occurrence features from different joints, persons, and frames. Then we construct aggregated features of multi-level co-occurrences (AFMC) from MC memory units to better represent the intra-frame action context encoding, and leverage a concurrent LSTM (Co-LSTM) to further model their temporal dynamics for action recognition. Experiments show that our proposed model significantly outperforms mainstream methods on NTU RGB+D 120 dataset and Northwestern-UCLA dataset.

Generalized anxiety disorder (GAD), as one of the most common chronic anxiety disorders, faces difficulties in clinical diagnosis. With the rapid development and wide application of smartphones in recent years, smartphones have a vivid application prospect in the field of mental disease monitoring and diagnosis. Based on WeChat applet platform on smartphones, an APP that integrates scale testing and inertial sensor data collection is developed to study the detection of subjects with GAD in task state. A behavior patterns extraction method is proposed using sliding windows to split behavior data, and processing data segments for clustering. Distribution information are extracted from the subjects' behavior patterns and are combined with the descriptive statistical features of the sample to identify GAD. The results show that this method has an accuracy of 66.44% for female subjects and 71.43% for male subjects in GAD recognition.

In the big data era, thousands of hundreds of devices play the role of data producer as well as data consumer. However, wireless devices bear the power exhausted problem in various situations. How to balance the trade-off between data processing speed and power efficiency is meaningful to be researched. In this study, we propose a Sleep Data Fusion Networks module (SFDN) which has a star topology Bluetooth network to fuse data of sleep-aware applications basing on our designed application protocol. In the network, a center Bluetooth device fuses data generated from target node Bluetooth devices. Due to the low power consumption of Bluetooth as well as connection safety, our method is a better choice for a long lifetime and non-real time sleep data processing and fusion tasks then the Wi-Fi-based approach used in EAST and Smart-Alarm sleep-aware applications.

Alzheimer's disease(AD) is a neurodegenerative disease that progresses slowly but worsens gradually, also, the most common kinds of dementia. Clinically, the diagnosis of AD is mainly based on rating scales and neuroimaging technology which is invasive, costly and time-consuming. Other than that, the clinical pathology has become irreversible when neuroimaging characteristics appear. It is imperative to develop new noninvasive methods for early diagnosis of AD. Several studies indicated the probable association of cognitive decline with gait changes might shed light on potential features for distinction of AD. This paper aims to exploit the feasibility of gait features for early diagnosis of mild cognitive impairment(MCI) and AD by using machine learning methods. A device-free AD detection system is built, with a natural undisturbed gait collecting system and a well-performed Long Short-Term Memory(LSTM) based model, in this article. Moreover, it can serve as a simplified, non-invasive, and highly accurate clinical auxiliary tool for early diagnosis and distinction of AD. Experimental results showed a 90.48%, 92.00%, and 88.24% in accuracy, sensitivity, and specificity respectively for distinguishing AD by using the method with LSTM based model. Furthermore, the gait cycle and stride length in MCI or AD were more variable than in healthy controls through redefining and calculating the gait features with skeleton data obtained by Kinect devices.

Gait-based person re-identification (Re-ID) is valuable for safety-critical applications, and using only 3D skeleton data to extract discriminative gait features for person Re-ID is an emerging open topic. Existing methods either adopt hand-crafted features or learn gait features by traditional supervised learning paradigms. Unlike previous methods, we for the first time propose a generic gait encoding approach that can utilize unlabeled skeleton data to learn gait representations in a self-supervised manner. Specifically, we first propose to introduce self-supervision by learning to reconstruct input skeleton sequences in reverse order, which facilitates learning richer high-level semantics and better gait representations. Second, inspired by the fact that motion's continuity endows temporally adjacent skeletons with higher correlations (“locality”), we propose a locality-aware attention mechanism that encourages learning larger attention weights for temporally adjacent skeletons when reconstructing current skeleton, so as to learn locality when encoding gait. Finally, we propose Attention-based Gait Encodings (AGEs), which are built using context vectors learned by locality-aware attention, as final gait representations. AGEs are directly utilized to realize effective person Re-ID. Our approach typically improves existing skeleton-based methods by 10-20% Rank-1 accuracy, and it achieves comparable or even superior performance to multi-modal methods with extra RGB or depth information.

The emerging 5G-enabled vehicular networks can satisfy various requirements of vehicles by traffic offloading. However, limited cellular spectrum and energy supplies restrict the development of 5G-enabled applications in vehicular networks. In this article, we construct an intelligent offloading framework for 5G-enabled vehicular networks, by jointly utilizing licensed cellular spectrum and unlicensed channels. A cost minimization problem is formulated by considering the latency constraint of users and is further decomposed into two subproblems due to its complexity. For the first subproblem, a two-sided matching algorithm is proposed to schedule the unlicensed spectrum. Then, a deep-reinforcement-learning-based method is investigated for the second one, where the system state is simplified to realize distributed traffic offloading. Real-world traces of taxies are leveraged to illustrate the effectiveness of our solution.

The public's purchase incentive increases dramatically during the holiday season and subsequently returns to normal levels. This seasonality is common in various scenarios and highlights the following questions: how does the public's purchase incentive fluctuate over the course of a year? Which factors are conducive to this seasonal behavior and how can they be modeled? In this paper, we propose a model that explicitly integrates temporal point process theory with the construction of a networked community, to describe the dynamics of collective action propagation with seasonal fluctuation. Furthermore, a database is constructed of sales records for 21 video game consoles and 13 237 video games in France, Germany, Japan, the U.K., the USA, and worldwide from 1989 to 2018. Experimental results suggest that peak desire always appears in the holiday season about one week before Christmas and is about four times higher than consumption desire in a normal period in all areas.

Efficient detection of outliers from massive data with a high outlier ratio is challenging but not explicitly discussed yet. In such a case, existing methods either suffer from poor robustness or require expensive computations. This paper proposes a Low-rank based Efficient Outlier Detection (LEOD) framework to achieve favorable robustness against high outlier ratios with much cheaper computations. Specifically, it is worth highlighting the following aspects of LEOD: (1) Our framework exploits the low-rank structure embedded in the similarity matrix and considers inliers/outliers equally based on this low-rank structure, which facilitates us to encourage satisfying robustness with low computational cost later; (2) A novel re-weighting algorithm is derived as a new general solution to the constrained eigenvalue problem, which is a major bottleneck for the optimization process. Instead of the high space and time complexity (O((2n)2)/O((2n)3)) required by the classic solution, our algorithm enjoys O(n) space complexity and a faster optimization speed in the experiments; (3) A new alternative formulation is proposed for further acceleration of the solution process, where a cheap closed-form solution can be obtained. Experiments show that LEOD achieves strong robustness under an outlier ratio from 20% to 60%, while it is at most 100 times more memory efficient and 1000 times faster than its previous counterpart that attains comparable performance. The codes of LEOD are publicly available at https://github.com/demonzyj56/LEOD.

Aiming to effective early detection of breast cancer, automatic tumor segmentation based on breast Magnetic Resonance Imaging (MRI) is concentrated by more and more researchers. This paper proposes a dense encoder-decoder network based on two-level context enhanced residual attention mechanism (TLCRAM-DED). With respect to TLCRAM-DED, we design the encoding structure combining two-level residual attention structure with dense block to extract and refine the features of different layers. Meanwhile, a dense multi-scale atrous convolution is used at the end of the encoder to obtain a larger receptive field and enrich the extracted semantic information. Moreover, residual attention structure (RAS) is also used for the refinement during decoding stage, while a long connection formed with the encoder RAS output is applied to supplement the features and to gradually recover the segmentation details. We validated prosed model in the DCE sequence of challenging breast cancer MRI dataset. The average Dice coefficient is up to 81.04%, which outperforms compared state-of-the-arts.

Body gesture, a crucial component of "body language", remains less explored to recognize emotion while face expression-based and speech-based approaches are widely investigated. In this paper, we introduce an exploratory experiment to recognize emotion using deep learning only from body gestures. 43,200 multi-view RGB videos of simplified body gestures and their neutral control groups are captured from 80 humans using Hikvision network cameras to support the experiment. A novel approach is proposed to use deep neural network fuse skeleton and RGB features only using single-modality RGB video data. Experimental results show our approach achieves substantial improvements both in individual categories and overall and is provided with stronger generalization capability as well.

Mobile edge computing (MEC) has risen as a promising paradigm to provide high quality of experience via relocating the cloud server in close proximity to smart mobile devices (SMDs). In MEC networks, the MEC server with computation capability and storage resource can jointly execute the latency-sensitive offloading tasks and cache the contents requested by SMDs. In order to minimize the total latency consumption of the computation tasks, we jointly consider computation offloading, content caching, and resource allocation as an integrated model, which is formulated as a mixed integer nonlinear programming (MINLP) problem. We design an asymmetric search tree and improve the branch and bound method to obtain a set of accurate decisions and resource allocation strategies. Furthermore, we introduce the auxiliary variables to reformulate the proposed model and apply the modified generalized benders decomposition method to solve the MINLP problem in polynomial computation complexity time. Simulation results demonstrate the superiority of the proposed schemes.

Unmanned aerial vehicle (UAV) has been witnessed as a promising approach for offering extensive coverage and additional computation capability to smart mobile devices (SMDs), especially in the scenario without available infrastructures. In this paper, a UAV-assisted mobile edge computing system with stochastic computation tasks is investigated. The system aims to minimize the average weighted energy consumption of SMDs and the UAV, subject to the constraints on computation offloading, resource allocation, and flying trajectory scheduling of the UAV. Due to nonconvexity of the problem and the time coupling of variables, a Lyapunov-based approach is applied to analyze the task queue, and the energy consumption minimization problem is decomposed into three manageable subproblems. Furthermore, a joint optimization algorithm is proposed to iteratively solve the problem. Simulation results demonstrate that the system performance obtained by the proposed scheme can outperform the benchmark schemes, and the optimal parameter selections are concluded in the experimental discussion.

The automated diagnosis helps us better understand the complex landscape of diseases, leading to more effective, early and reliable medical diagnosis and therapy. The human-machine interactions in healthcare delivery relying on automated cyber-physical systems (ACPSs) play an important role in the automated diagnosis. Currently, the multicore accelerator used for ACPS has utilized the network-on-chip (NoC) for personalized healthcare. However, the discrete cores based on NoC are affected by limited computation speed, since the data have to pass through an electrical interconnect. In this paper, we propose a novel optical NoC (ONoC) solution of designing discrete cores to quickly understand biomarkers for early detecting abnormal pathophysiology, such as the deviation from the protein's native state. We analyze the performance of our ONoC-based ACPS accelerator for personalized healthcare by virtue of the tested proteins widely adopted in the lattice protein model. Our mathematical analysis and simulation results demonstrate that: 1) the chip area becomes smaller than a traditional design, which makes the personalized healthcare product more convenient; 2) the computation speed is promoted, resulting in the rapid understanding of biomarkers; and 3) we improve the data transmission reliability through accurately capturing the photonic effect so that desirable human-machine interactions can be guaranteed. Note to Practitioners-We design an on-chip hardware accelerator for automated diagnosis and personalized healthcare by predicting biological protein folding. The simulation results based on the lattice protein model can well guide the practitioners to design a more convenient and reliable product quickly detecting the biomarker, such as the deviation from the protein's native state.

Currently, vehicles have the abilities to communicate with each other autonomously. For Internet of Vehicles (IoV), it is urgent to reduce the latency and improve the throughput for data transmission among vehicles. This article proposes a deep learning based transmission strategy by exploring trirelationships among vehicles. Specifically, we consider both the social and physical attributes of vehicles at the edge of IoV, i.e., edge of vehicles. The social features of vehicles are extracted to establish the network model by constructing triangle motif structures to obtain primary neighbors with close relationships. Additionally, the connection probabilities of nodes based on the characteristics of vehicles and devices can be estimated, by which a content sharing partner discovery algorithm is proposed based on convolutional neural network. Finally, the experiment results demonstrate the efficiency of our method with respect to various aspects, such as message delivery ratio, average latency, and percentage of connected devices.

In many people's daily life, alarm sounds play an important role, which reflects the fast paced life in the modern society. On most occasions, people uses alarm sounds to wake them up in the morning. Improper alarm sounds could make people feel terrible. In this paper, we mainly propose a smart alarm sound recommendation system and construct an application to study how alarm sounds can impact human emotions. The recommendation system is deployed on the cloud, working with smartphones to deliver smart alarm sounds by considering not only sleep patterns, but also context information such as weather. The designed system can recommend smart alarm sounds to users, orchestrate sensing data collected by multiple sensors on smartphones, and collaborate with cloud computing to recommend preferable alarm sounds. An application is developed to demonstrate system effectiveness, which consists of the fore-end on Android OS and the back-end on the cloud. Experiments demonstrate that our system can recommend smart alarm sounds to wake participants up in the morning and the participants give feedback about their emotional states. The results show the system can improve people's emotion states by about 14.57%, compared to traditional alarm sound delivery.

Driving is an important part in daily life. But it is also dangerous when people drive in a bad mood or when people are tired. In this paper, we propose SAR, a novel social aware route recommendation system, which is designed to recommend proper routes to drivers to help diminish negative mood and fatigue and help improve driving experience. SAR integrates multiple sources of sensing data from smartphones to recommend a proper route to the driver considering the driver’s real-time mood-fatigue, history record, and social information. Experiments have been conducted with SAR to demonstrate the system’s effectiveness as well as reasonable computation and communication overheads on smartphones.

Because of the enormous potential to guarantee road safety and improve driving experience, social Internet of Vehicle (SIoV) is becoming a hot research topic in both academic and industrial circles. As the ever-increasing variety, quantity, and intelligence of on-board equipment, along with the evergrowing demand for service quality of automobiles, the way to provide users with a range of security-related and user-oriented vehicular applications has become significant. This paper concentrates on the design of a service access system in SIoVs, which focuses on a reliability assurance strategy and quality optimization method. First, in lieu of the instability of vehicular devices, a dynamic access service evaluation scheme is investigated, which explores the potential relevance of vehicles by constructing their social relationships. Next, this work studies a trajectory-based interaction time prediction algorithm to cope with an unstable network topology and high rate of disconnection in SIoVs. At last, a cooperative quality-aware system model is proposed for service access in SIoVs. Simulation results demonstrate the effectiveness of the proposed scheme.

As an emerging communication platform, opportunistic Cloud of Things (CoT) is promising for clients to exchange messages through opportunistic contacts in cloud computing-enabled Internet of Things. Recently, numerous socially aware schemes have been put forward, leveraging users’ social attributes and contact history to predict future contacts with the purpose of improving message forwarding efficiency and network throughput. However, individual privacy is generally overlooked in the prediction process and transmission stage of opportunistic CoT. In this paper, we construct a privacy-preserving message forwarding framework for opportunistic CoT to guarantee individual privacy and improve transmission efficiency. We first set up a two-layer architecture of a cloud server to improve communication efficiency for terminal clients. By integrating a security-based mobility prediction algorithm with a routing decision process, our scheme can effectively protect individual privacy. We integrate an attribute-based cryptographic algorithm with a message delivery process to enable our scheme to resist attacks, such as Sybil attack, drop for profit, and data tampered attack. Compared with some existing solutions, our scheme improves network security significantly at the cost of slightly increased communication overhead.

Mobile edge computing (MEC) brings computation capacity to the edge of mobile networks in close proximity to smart mobile devices (SMDs) and contributes to energy saving compared with local computing, but resulting in increased network load and transmission latency. To investigate the tradeoff between energy consumption and latency, we present an energy-aware offloading scheme, which jointly optimizes communication and computation resource allocation under the limited energy and sensitive latency. In this paper, single and multicell MEC network scenarios are considered at the same time. The residual energy of smart devices' battery is introduced into the definition of the weighting factor of energy consumption and latency. In terms of the mixed integer nonlinear problem for computation offloading and resource allocation, we propose an iterative search algorithm combining interior penalty function with D.C. (the difference of two convex functions/sets) programming to find the optimal solution. Numerical results show that the proposed algorithm can obtain lower total cost (i.e., the weighted sum of energy consumption and execution latency) comparing with the baseline algorithms, and the energy-aware weighting factor is of great significance to maintain the lifetime of SMDs.

Sharing economy refers to peer-based activities of obtaining, giving, or sharing the access to goods and services, coordinated through community-based online services. It is known as collaborative consumption that people share the services rather than having individual ownership. By leveraging idle resources to produce more goods and services, sharing economy significantly drives green consumption and sustainable development in our human society. Using information technology to provide individuals with information enables the optimization of resources through the mutualization of excess capacity in goods and services. A common premise is that when information is shared, the value of the goods may increase for businesses, for individuals, for communities, and for the whole society in general. Currently, sharing economy has potentially resulted in a great impact on citizens’ everyday life and generated huge economic benefits, e.g., Airbnb, Uber, and Amazon Mechanical Turk. A host of enabling technologies has reached the mainstream for the rise of sharing economy, including open data, the ubiquity of low-cost mobile phones, and social media. These technologies dramatically reduce the friction of share-based business and organizational models.

Object detection is important in car sharing services. Accuracy, efficiency, and low memory consumption are desirable for object detection in car sharing services. This paper presents a network system that satisfies all these requirements. Our approach first divides the object detection task into multiple simpler local regression tasks. Then, we propose the generalized Haar filter-based convolutional neural network to reduce the consumption of memory and computing resource. To achieve real-time performance, we introduce a sparse window generation strategy to reduce the number of input image patches without sacrificing accuracy. We perform experiments on both vehicle and pedestrian data sets. Experimental results demonstrate that our approach can accurately detect objects under challenging conditions. Note to Practitioners-Object detection is an important part of intelligent vehicle technologies, which play an important role in car sharing services. Object detection provides metadata for collision avoidance, self-driving systems, and driver-assistance systems, which can result in better safety and consumer experiences in car sharing services. Although deep learning has achieved an excellent performance in object detection, they consume a large amount of storage and computing resource, which makes them difficult to be deployed for car sharing services. This paper suggests a novel approach which is based on the generalized Haar filter and the local regression strategy. Our approach is accurate, efficient, and light. The experimental results verify the effectiveness of the proposed approach in car sharing services.

People spend up to one-third of lives asleep, and healthy sleep habits can make a big difference in their quality of life. But in modern society, many people have unhealthy sleep diaries and suffer from various sleep disorders, which may result in irregular mood fluctuations or even mental health problems such as anxiety and depression. We propose the Emotion-Aware Smart Tips (EAST), a novel approach that could help to inform users about their irregular emotional states with smart tips to improve their sleep qualities. EAST aims at helping users keep healthy sleep schedules and emotional states by providing smart tips through a novel model that combines multivariate regression, random forest, and neural network to quantify the relations between sleep patterns and emotional states. Prototype implementation and initial experiments of EAST in mobile phones have demonstrated its desired functionality and practicality for real-world deployment.

Stable and reliable wireless communication is one of the critical demands for smart cities to connect people and devices. Although intelligent terminals can be leveraged to deliver and exchange data through Internet, poor network coverage and expensive network access challenge the deployment of network infrastructure. In this paper, we propose a social-oriented smartphone-based adaptive transmission mechanism to improve the network connectivity and throughput in Internet of Things (IoTs) for smart cities. First, a social-oriented double-auction-based relay selection scheme is investigated to stimulate the relay smartphones to forward packets for others so that the network connectivity can be strengthened. Furthermore, for the sake of achieving high throughput in smartphone-based IoTs, the relay method selection is determined by integrating various kinds of transmission schemes in an optimal fashion to make full use of wireless spectrum resource. Due to its high computational complexity, a firefly-algorithm-based scheme is investigated, by which the formulated NP-complete problem can be solved effectively. Simulation results demonstrate the superiority of our proposed method.

This paper proposes a novel power management solution for resource-constrained devices in the context of Internet of Things (IoT). We focus on smartphones in the IoT, as they are getting increasingly popular and equipped with strong sensing capabilities. Smartphones have complex and asynchronous power consumption incurred by heterogeneous components including their on-board sensors. Their interaction with the cloud allows them to offload computation tasks and access remote data storage. In this work, we aim at monitoring the power consumption behaviours of the smartphones, profiling both individual applications and the system as a whole, to make better decisions in power management. We design a cloud orchestration architecture as an epic predictor of behaviours of smart devices by extracting their application characteristics and resource utilization. We design and implement this architecture to perform energy profiling and data analysis on massive data logs. This cloud orchestration architecture coordinates a number of cloud-based services and supports dynamic workflows between service components, which can reduce energy consumption in the energy profiling process itself. Experimental results showed that small portion of applications dominate the energy consumption of smartphones. Heuristic profiling can effectively reduce energy consumption in data logging and communications without scarifying the accuracy of power monitoring.

The blind rendezvous, which requires neither Common Control Channel (CCC) nor the information of the target user's available channels, has recently attracted a lot of research interests. As a contribution to this research area, in this paper we propose a Sender-Jump Receiver-Wait blind rendezvous algorithm, which has fully satisfied the following requirements: 1) guaranteeing rendezvous; 2) realizing full rendezvous diversity, i.e., any pair of users can rendezvous on all commonly available channels; 3) requiring no time-synchronization; 4) supporting both symmetric and asymmetric models; 5) supporting multi-user/multi-hop scenarios and 6) consuming short Time-to-Rendezvous (TTR). Theoretical analysis and computer simulations have validated our algorithm.

Driving is an integral part of our everyday lives, and the average driving time of people globally is increasing to 84 minutes everyday, which is a time when people are uniquely vulnerable. A number of research works have identified that mobile crowd sensing in vehicular social networks (VSNs) can be effectively used for many purposes and bring huge economic benefits, e.g., safety improvement and traffic management. This paper presents our effort that toward context-aware mobile crowd sensing in VSNs. First, we introduce a novel application-oriented service collaboration (ASCM) model which can automatically match multiple users with multiple mobile crowd sensing tasks in VSNs in an efficient manner. After that, for users' dynamic contexts of VSNs, we proposes a context information management model, that aims to enable the mobile crowd sensing applications to autonomously match appropriate service and information with different users (requesters and participants) in crowdsensing.

Millions of people are severely injured or killed in road accidents every year and most of these accidents are caused by human error. Fatigue and negative emotions such as anger adversely affect driver performance, thereby increasing the risk involved in driving. Research has shown that listening to the right kind of music in these situations can ameliorate driver performance and improve road safety. Context-aware music delivery systems succeed in delivering suitable music according to the situation through the process of music mood-mapping which identifies the mood of a song. Additionally, we can leverage the power of the cloud to enable crowd sensing of the mood-mapping of various songs and enhance the effectiveness of situation-aware music delivery for drivers. The cloud can be used to aggregate the crowd sensed music mood-mapping data and improve the effectiveness of music delivery by providing accurate mood-mappings from the aggregated data. Currently, context-aware music delivery systems consider only features from the song for music mood-mapping. In this paper, we propose a novel approach to music mood-mapping for drivers which also incorporates the social context of a driver including age, gender and cultural background to enhance the effectiveness of music delivery in context-aware music recommendation systems for drivers.

This paper proposes a novel power management solution for resource-constrained devices in the context of Internet of Things (IoT). We focus on smartphones in the IoT, as they are getting increasingly popular and equipped with strong sensing capabilities. Smartphones have complex and asynchronous power consumption incurred by heterogeneous components including their on-board sensors. Their interaction with the cloud allows them to offload computation tasks and access remote data storage. In this work, we aim at monitoring the power consumption behaviours of the smartphones, profiling both individual applications and the system as a whole, to make better decisions in power management. We design a cloud orchestration architecture as an epic predictor of behaviours of smart devices by extracting their application characteristics and resource utilization. We design and implement this architecture to perform energy profiling and data analysis on massive data logs. This cloud orchestration architecture coordinates a number of cloud-based services and supports dynamic workflows between service components, which can reduce energy consumption in the energy profiling process itself. Experimental results showed that small portion of applications dominate the energy consumption of smartphones. Heuristic profiling can effectively reduce energy consumption in data logging and communications without scarifying the accuracy of power monitoring.

In this paper, we present a mobility cloud platform called CarCloud that has been designed to facilitate the real-world deployment of ubiquitous mobile telematic applications to promote smart transportation. CarCloud leverages the mobility, sensing and communication capacities of mobile devices, and collaborates with sensors embedded in vehicles (e.g., accessed via OBD-II) and cloud services to form a seamless platform. This platform orchestrates different mobility data in transportations to REST web service based mobility services that could be used for different mobile telematic applications (e.g., enabling traffic managers to monitor the behaviors of drivers) in intelligent transportation systems (ITS). Prototype implementation and preliminary experiments demonstrate the desired functionality of CarCloud for ITS and its feasibility for real-world deployment.

It is predicted that billions of intelligent devices and networks, such as wireless sensor networks (WSNs), will not be isolated but connected and integrated with computer networks in future Internet of Things (IoT). In order to well maintain those sensor devices, it is often necessary to evolve devices to function correctly by allowing device management (DM) entities to remotely monitor and control devices without consuming significant resources. In this paper, we propose a lightweight RESTful Web service (WS) approach to enable device management of wireless sensor devices. Specifically, motivated by the recent development of IPv6-based open standards for accessing wireless resource-constrained networks, we consider to implement IPv6 over low-power wireless personal area network (6LoWPAN)/routing protocol for low power and lossy network (RPL)/constrained application protocol (CoAP) protocols on sensor devices and propose a CoAP-based DM solution to allow easy access and management of IPv6 sensor devices. By developing a prototype cloud system, we successfully demonstrate the proposed solution in efficient and effective management of wireless sensor devices.

Driving is an integral part of our everyday lives, but it is also a time when people are uniquely vulnerable. Previous research has demonstrated that not only does listening to suitable music while driving not impair driving performance, but it could lead to an improved mood and a more relaxed body state, which could improve driving performance and promote safe driving significantly. In this article, we propose SAfeDJ, a smartphone-based situation-aware music recommendation system, which is designed to turn driving into a safe and enjoyable experience. SAfeDJ aims at helping drivers to diminish fatigue and negative emotion. Its design is based on novel interactive methods, which enable in-car smartphones to orchestrate multiple sources of sensing data and the drivers' social context, in collaboration with cloud computing to form a seamless crowdsensing solution. This solution enables different smartphones to collaboratively recommend preferable music to drivers according to each driver's specific situations in an automated and intelligent manner. Practical experiments of SAfeDJ have proved its effectiveness in music-mood analysis, and mood-fatigue detections of drivers with reasonable computation and communication overheads on smartphones. Also, our user studies have demonstrated that SAfeDJ helps to decrease fatigue degree and negative mood degree of drivers by 49.09% and 36.35%, respectively, compared to traditional smartphone-based music player under similar driving situations.

The powerful data storage and data processing abilities of cloud computing (CC) and the ubiquitous data gathering capability of wireless sensor network (WSN) complement each other in CC-WSN integration, which is attracting growing interest from both academia and industry. However, job scheduling for CC integrated with WSN is a critical and unexplored topic. To fill this gap, this paper first analyzes the characteristics of job scheduling with respect to CC-WSN integration and then studies two traditional and popular job scheduling algorithms (i.e., Min-Min and Max-Min). Further, two novel job scheduling algorithms, namely priority-based two phase Min-Min (PTMM) and priority-based two phase Max-Min (PTAM), are proposed for CC integrated with WSN. Extensive experimental results show that PTMM and PTAM achieve shorter expected completion time than Min-Min and Max-Min, for CC integrated with WSN.

Driving is an integral part of our everyday lives, but it is also a time when people are uniquely vulnerable. Poor road condition, traffic congestion and long driving time may bring negative emotion to drivers and increase the chance of traffic accidents. We propose SAfeDJ, a situation-aware in-car music delivery application, which turns people's trips into pleasant journeys and driving into a safe and enjoyable activity. SAfeDJ aims at helping drivers to diminish fatigue and negative emotion. It is built on a vehicular healthcare platform that enables communications among drivers and integrates with multiple types of sensors to promote safe driving. Prototype implementation and initial results of SAfeDJ have demonstrated its desired functionality in drivers' daily lives and feasibility for real-world deployment.