Federated learning (FL), as a newly-developing technique, brings the advantage of organizing multiple participants to learn together, while avoiding the leakage of their privacy information. Contract theory provides an effective incentive mechanism to encourage participants to participate in FL. Existing contract-based incentive mechanisms consider participants’ types but ignore the different contributions of participants within the same type during the training.This paper first introduces a metric, reputation, to evaluate the contribution of participants in each iteration, and then proposes a hybrid contract mechanism consisting of a short-term contract and a long-term contract. Only the participants with reputations higher than a pre-defined threshold can sign the long-term contract. We formulate the solution of the long-term contract mechanism as an optimization problem with constraints. We further simplify the constraints of the long-term contract optimization problem, and theoretically analyze the correctness of the simplification to greatly reduce its computational complexity. We prove that the model owner achieves more profit with the hybrid contract mechanism. Simulations with the MNIST dataset show that the long-term contract improves the model accuracy by at least 5% compared with the existing contracts. Furthermore, compared with the short-term contract, participants signing the long-term contract are granted more rewards.

As an emerging paradigm, orbital edge computing (OEC) has garnered significant attention due to its advantages in global coverage and seamless connectivity. Most existing research adopts a user-centric approach, focusing on enhancing user satisfaction, but overlooks the economic objectives of satellites. To fill this gap, this paper investigates the collaborative optimization of computing resource pricing, task offloading, and resource allocation, aiming to maximize satellite profits while ensuring fair resource distribution to meet user computing demands. Firstly, a bilevel optimization problem (BOP) is established, considering the coupling between satellite and user strategies. In the upper level, satellites determine unit prices to maximize total profit, while in the lower level, users decide offloading modes and required computing resource amounts based on given prices. Moreover, by deriving the relationship between user resource demands and offloading modes, the mixed variable optimization in the original BOP’s lower level is transformed into a discrete variable optimization. To address the transformed BOP, we reduce the search space through server pruning and design a nested optimization algorithm, which utilizes particle swarm optimization (PSO) and matching game theory in upper and lower levels, respectively. Lastly, experiments prove our algorithm surpasses state-of-the-art in satellite total profit and user task completion rate.

Sign language is a basic form of communication for hearing-impaired individuals. An evaluation of the quality of sign language gestures helps improve the efficiency of sign language learning. This article proposes SignEvaluator, a sign language quality assessment system with a movement quality feature extractor and assessment generator. In the former, three quality measures are proposed for gestures and sentences. The trajectory of the palm is mapped onto position space with kernel density estimation. For finger movements, the instantaneous energy and curvature of the gesture signals are extracted with Bézier curves. Meanwhile, the performer's familiarity with gestures is indicated by the movement fluency metric of sentences. In the assessment generator, the final assessment results are calculated by combining the weights of different quality metrics and the confidence of different gesture levels. The results indicate that SignEvaluator obtained an F1-score of 0.89 for 702 sentences collected from 20 performers.

Sign language provides communication support for deaf and severely hearing-impaired people. Sign language translation (SLT) bridges the hearing-impaired and hearing communities. Existing SLT methods use gloss as intermediate supervisory information to help the model sense gesture boundaries and understand global semantics. However, annotating gloss requires great cost, especially in multimodal SLT task. This paper proposes GFTLS-SLT: gloss-free Transformer based lexical and semantic awareness framework for SLT. The multimodal alignment and fusion module in GFTLS-SLT utilizes cross-attention to align multimodal features, and fuses them using the improved statistical and contrastive attention. To replace the role of gloss, GFTLS-SLT designs gesture lexical awareness (GLA) and global semantic awareness (GSA) modules. The GLA module utilizes the defined observation matrix to obtain the lexical meaning matrix, and makes the model sense gesture boundaries by the designed dynamic step-size lexical matching algorithm. The multimodal semantic header is used by GSA module to represent the sign language global semantic and is aligned with the spoken semantic on semantic space. In addition, the experiment results of GFTLS-SLT on publicly available multimodal SLT datasets show that its performance reaches that of SLT methods with gloss supervision.

Whole-body mesh reconstruction utilizes neural networks to reconstruct the 3D human body, face, and hands, forming a fundamental task in computer vision. It is used to model human action in many practical applications that prioritize upper body action, particularly the hands. However, accurately estimating the 3D mesh parameters of hands in practical applications remains challenging due to severe self-occlusion and high self-similarity in hand action. To address these challenges, the Accurate Hand Modeling in Whole-Body Mesh Reconstruction (AHM-WBMR) is proposed in this article. It mainly consists of two innovative components: the joint-level features progressive matching and refinement and the kinematic features propagation. In the joint-level features progressive matching and refinement, the 3D deformable cross attention and the 3D deformable transformer decoder are proposed to assist in refining hand joint-level features. Further, in the kinematic features propagation, the kinematic-aware topology network is proposed to distinguish and relate different hand joint-level features using three types of kinematic topology structures. We evaluated AHM-WBMR on the UBody and FreiHAND datasets, both of which contain rich hand movements. Compared with the state-of-the-art methods, we achieved improvements of at least 10.2% and 11.1% in hand-related metrics on the two datasets, respectively.

Emotional expression is a bridge to human communication, especially for the hearing impaired. This paper proposes a sign language emotion recognition method based on semantic and movement features by exploring the relationship between emotion valence and arousal in-depth, called SeMER. The SeMER framework includes a semantic extractor, a movement feature extractor, and an emotion classifier. The contextual relations obtained from the sign language recognition task are added to the semantic extractor as prior knowledge using a transfer learning approach to better acquire the affective polarity of semantics. In the movement feature extractor based on graph convolutional networks, a spatial-temporal adjacency matrix of gestures and node attention matrix are developed to aggregate the emotion-related movement features of intra- and inter-gestures. The proposed emotion classifier maps semantic and movement features to the emotion space. The validated U-shaped distributions of valance and arousal are then used to guide the relationship between them, and improve the accuracy of emotion prediction. In addition, a sign language emotion dataset containing 5 emotions from 18 participants, SE-Sentence, is collected through armbands with built-in surface electromyograph and inertial measurement unit sensors. Experimental results showed that SeMER achieved an accuracy and f1 value of 88% on SE-Sentence.

This paper studies the access control problem of long-term throughput maximization in wireless communication systems with Energy Harvesting (EH). In the existing research, many access schemes based on accurate environmental information have been proposed, such as channel information and the EH process. However, access to environmental information is costly, and traditional access control frameworks are expensive to explore in high-dimensional spaces. Thus, an access control framework based on hierarchical reinforcement learning (HRL) is proposed in this paper. In HRL, the control problem in the Markov decision process (MDP) form is decomposed into a multilevel sequential control problem. It includes high-level channel number selection, mid-level channel selection, and low-level channel matching subproblems. The scheme is obtained by combining the solutions of subproblems at different level which are solved in sequence. In addition, to improve learning efficiency, the deterministic action (DA) module and the prior knowledge (PK) module are put forward. The DA module solves the channel matching problem under the additional guidance given by the previous subproblem, which selects definite good low-level actions. The PK module provides the framework with the common knowledge of the system structure learned from the hypothetical environment, so as to obtain better initial performance. Experimental results show that our framework achieves better performance and better learning efficiency compared with several recent transmission schemes. Note to Practitioners—Access control is an important issue in wireless communication systems, and users need to be scheduled to solve the constraint of limited resources, such as energy usually provided by batteries. In recent years, in order to overcome the energy limitation, energy harvesting devices have been developed and applied to wireless communication systems. However, the energy collection ability of the system is greatly influenced by the environment, which leads to the poor performance of most traditional control schemes that rely on the prior knowledge of the environment. Therefore, this paper proposes a novel hierarchical reinforcement learning (HRL)-based model-free access control framework for wireless communication system to maximize the system throughput without any prior environmental knowledge. The scheme abstracts the original control problem into three sub-control sub control problems according to tasks and solves them sequentially, thus simplifying the original control problem. This scheme can not only learn independently, but also does not depend on the prior knowledge of the environment. Moreover, this method is also suitable for the large-scale environment while the conventional end-to-end reinforcement learning is not suitable for. Compared with traditional algorithms, our method has better performance and higher learning efficiency.

Sign language is widely used among deaf. Many existing studies on Sign Language Recognition (SLR) focus on addressing communication barriers between deaf and hearing people. However, current studies face two challenges: the recognition result is highly dependent on the wearable device placements, and the existence of sentences in the training set. To address the challenges, this paper proposed EasyHear – a Transformer-based Chinese SLR system with a generalization approach. For generalization of wearable device placements, an anchor-based signal rotation correction algorithm is proposed to eliminate the impact of variations in wearing positions. In terms of generalization of sentences, a gesture code is constructed to reflect the closeness of gestures after defining an intimate entropy of gestures. Moreover, a semantic code is developed by training a neural network on a large corpus to reflect the intimacy of Chinese Sign Language (CSL) words on unseen sentences in the training set. A Transformer-based model combined with the rotated gesture signals as well as gesture and semantic codes is suggested to improve recognition performance across various wearing positions and sentences. The results show that EasyHear achieved an average word error rate of 21.60% for samples of 712 commonly used CSL sentences.

Sign language is the basic way for people with hearing impairment to communicate, and sign language recognition (SLR) could effectively help in this regard. Mainstream Transformer-based SLR requires positional encoding to sense the positional information of the data. However, existing PE methods globally encode the sign data result in weaken or even ignoring the sequence variation within the gestures. This paper proposes HDTSLR: A Transformer-based SLR framework built on hierarchical dynamic positional encoding (HDPE) enhances individual gesture sequence features while preserving the sign overall temporal features. HDPE designs semantic positional encoding utilizing predefined scale functions with trainable biases to emphasize sign semantic relationships. The t-distribution is used by the designed lexical positional encoding to explore the unique variation of gestures. Before the HDPE operation, the sign language data is split into equal-length feature clips while feature extraction and chunking are performed by the autoencoder. The feature clips with significant changes in gesture chunk are further selected and aggregated with the remaining ones by deforming Gram matrix. In addition, HDTSLR is evaluated on the one-handed and two-handed datasets, achieving word error rates of 16.59% and 21.67%, respectively. Comparison experiments show that it outperforms known SLR methods in both accuracy and robustness.

Sign language recognition (SLR) enables hearing-impaired people to better communicate with able-bodied individuals. The diversity of multiple modalities can be utilized to improve SLR. However, existing multimodal fusion methods do not take into account multimodal interrelationships in-depth. This paper proposes SeeSign: a multimodal fusion framework based on statistical attention and contrastive attention for SLR. The designed two attention mechanisms are used to investigate intra-modal and inter-modal correlations of surface Electromyography (sEMG) and inertial measurement unit (IMU) signals, and fuse the two modalities. Statistical attention uses the Laplace operator and lower quantile to select and enhance active features within each modal feature clip. Contrastive attention calculates the information gain of active features in a couple of enhanced feature clips located at the same position in two modalities. The enhanced feature clips are then fused in their positions based on the gain. The fused multimodal features are fed into a Transformer-based network with connectionist temporal classification and cross-entropy losses for SLR. The experimental results show that SeeSign has accuracy of 93.17% for isolated words, and word error rates of 18.34% and 22.08% on one-handed and two-handed sign language datasets, respectively. Moreover, it outperforms state-of-the-art methods in terms of accuracy and robustness.

Sign language recognition (SLR) can connect the hearing-impaired and able-bodied communities. The SLR works through multiple modalities of co-action, which has garnered attention. However, these methods are much less effective or even fail in recognition when confronted with missing modalities. Therefore, this paper proposes MMSLR, a multimodal SLR framework with cross-modal complementary information. The framework comprises three key components: the cross-modal information complementation (CMIC) module, the fusion and prediction module (FPM), and the sign language recognition module (SLRM). The CMIC module is designed with multilayer, multi-view spatial-temporal detectors to observe different modality features in both temporal and spatial dimensions. Additionally, it utilizes co-training to achieve complementary information among multi-modalities. The FPM integrates crossmodal attention with Canberra distance to eliminate inter-modal redundant information while fusing multimodal features. The SLRM constructed based on Transformer fuses partially obtained modalities from CMIC through bidirectional cross-channel attention. Teacher-Student pairs are constructed to transfer fullmodal features from FPM to the above fused modality features. Moreover, experimental results on the provided MM-Sentence and publicly available OH-Sentence, TH-Sentence and USTCCSL datasets demonstrate that MMSLR achieves state-of-the-art performance.

Sign language serves as a bridge between the hearing-impaired and other people. Existing sensor-based approaches tend to only collect data from the dominant hand. Does this signal collection method affect the accuracy of gesture recognition, especially gestures where the dominant hand has the same movement while the non-dominant hand has different movements? The specific gestures are called same-right-and-different-left (SRDL) where the right hand is dominant. This article is the first to propose an SRDL-aware sign language recognition system. First, an SRDL discriminator based on an autoencoder and range classifier is designed to determine whether the gesture is SRDL. Second, an SRDL feature selector based on clustering relationship is presented. Multivariate variational mode decomposition and fast fourier transform are used to obtain the feature expression. Moreover, a clustering relationship algorithm is proposed to dynamically select features for every group of SRDL gestures in the feature expression. Finally, the experimental results show that the average word error rate is 14.3% and decreases by 8.5% and 12.1% compared with Signspeaker and MyoSign, respectively.

Understanding sign gestures is an essential step to helping individuals with hearing impaired. The existing works can only identify a small set of gestures accurately and the accuracy rate drops sharply with an increasing number of gestures. Because there are two challenges—a large number of similar gestures in sign language and the various signing speed of different people. Based on commercial smart bracelets, this article proposes a large-vocabulary sign language recognition system (which we call L-sign). First, we propose an entropy-based forward and backward matching algorithm to segment each gesture signal. Second, we design a gesture recognizer including a candidate gesture generator and semantic-based voter. The candidate gesture generator is aimed at providing candidate gesture designs based on a 3-branch convolutional neural network. The purpose of a semantic-based voter is to select the target gesture from candidate gestures by scoring, where the semantic distances between the last gesture in the current sentence and any candidate gestures is calculated, and a multilayer k-means algorithm is proposed to obtain a multilayer sign word structure to complete the scores of candidate gestures. Lastly, we deployed L-sign on the MYO bracelet. For 200 commonly used Chinese sign gestures, the experimental results show that the average accuracy rate was greater than 90%.

With the development of microelectronics and sensor technologies, there are more and more researchers applying them to human action recognition, most of which are professional motion and rely on specific-designed sensors and wearable de-vices. Meanwhile, the need of rehabilitation training is increasing due to occupational diseases, bad life-style and incorrect exercise habit. However, it is costly and inconvenient to train in clinics and hospitals. To buy or borrow a set of medical training equipment is also unpractical. In this paper, we propose to use smart phones, which have larger computing power and are equipped with richer sensors ever than before, to run artificial intelligence based models and algorithms for identification of rehabilitation actions. Beyond doubt, it will be more convenient to use smart phones instead of professional equipments. Nevertheless, there are still some challenges which prevent it from being put into practice, such as phone deployment, data collection, and model training. We initially conceptualize and implement a smart phone-based accuracy judgment system for rehabilitation action. According to the characteristics of the system, e.g., sensor difference, position variation, and computing power limitation, a supervised and data-sharing learning algorithm is proposed, the operation framework, loss function and regular expression function are carefully selected. The experiment on a prototype of the system verifies that the proposed method precisely identifies the rehabilitation actions of testees.

Facial micro-expressions are characterized by their extremely short duration and low intensity, can provide an important basis for judging people's emotions, and therefore have promising potential applications in numerous fields. This paper puts forward a novel method for recognizing microexpressions by using a quaternion-based sparse representation (QSR) model combined with the integral projection of difference energy image (IP-DEI) to extract features from color images of human faces . Using the quaternion model to jointly process color images can obtain greater feature information than gray or RGB images, and the QSR model helps reduce feature dimensions and enables greater discriminative representation. First, each microexpression sample undergoes IP-DEI to allow the features of all samples to be displayed in the form of a quaternion matrix Y. Next we find overcomplete dictionary matrix D and sparse coefficient matrix X such that Y = DX in ideal scenarios, and consider X̂̃̅̅̆̆̇ to be the features contained within the microexpression samples. Finally, we apply our method to the SMIC, CAMSE I and CAMSE II micro-expression databases while using SVM as classifier. The results of the experiment demonstrate that our method outperformed the currently most advanced methods in terms of micro-expression recognition accuracy.

In some specific applicable scenarios, nodes are placed in some geographical areas and limited to move in their own community. We investigate the tradeoff between the delivery delay and the number of transmissions in the above multi-community delay tolerant networks by propagating a data packet in a carefully chosen segment of community. First, three restricted epidemic routings are proposed: the shortest community-hop path scheme, the rectangle scheme, and the parallelogram scheme. Second, the ratios of the average number of communities that can propagate the data packet in the proposed schemes to those that can propagate in the epidemic routing are analyzed. The ratios are found to be small and to decrease with the increase in the number of communities. The tail distribution of the inter-meeting time of any two nodes in the neighboring communities is then demonstrated to be exponential. Third, the delivery delay of the proposed schemes is analyzed by Markovian chain tool. The experiments show that the theoretical model proposed here is reliable, and that the proposed schemes can significantly decrease the number of transmissions, even if these schemes increase the delivery delay to some extent.

Routing is one of the most challenging aspects in Delay Tolerant Networks (DTN) because the end-to-end path does not always occur. In wildlife tracking, habitat monitoring, and other scenarios, the network area may be divided into some geographical communities, the nodes only move in their communities. In this paper, for this specific community-based DTN, we address the delivery delay of the epidemic routing. The inter-meeting time of two nodes in adjacent communities is proven to be an exponential distribution. Moreover, the delivery delay of each community is obtained using Markovian chain mathematical tool when the epidemic routing is applied. The simulation confirms that our theoretical results fit the simulation results well.

A wireless ad hoc network consists of mobile nodes that are powered by batteries. The limited battery lifetime imposes a severe constraint on the network performance, energy conservation in such a network thus is of paramount importance, and energy efficient operations are critical to prolong the lifetime of the network. All-to-all multicasting is one fundamental operation in wireless ad hoc networks, in this paper we focus on the design of energy efficient routing algorithms for this operation. Specifically, we consider the following minimum-energy all-to-all multicasting problem. Given an all-to-all multicast session consisting of a set of terminal nodes in a wireless ad hoc network, where the transmission power of each node is either fixed or adjustable, assume that each terminal node has a message to share with each other, the problem is to build a shared multicast tree spanning all terminal nodes such that the total energy consumption of realizing the all-to-all multicast session by the tree is minimized. We first show that this problem is NP-complete. We then devise approximation algorithms with guaranteed approximation ratios. We also provide a distributed implementation of the proposed algorithm. We finally conduct experiments by simulations to evaluate the performance of the proposed algorithm. The experimental results demonstrate that the proposed algorithm significantly outperforms all the other known algorithms.

This paper focuses on network coding based multirate multimedia streaming in directed networks and aims at maximizing the total layers received by all receivers, which directly determine the quality of video streaming. We consider the property of layered coding in video streaming and propose the layer separated network coding scheme (LSNC) for layered video streaming. Two algorithms OLSNC and SLSNC are proposed for LSNC based video streaming, where OLSNC achieves an optimal solution, while SLSNC is a polynomial time approximation algorithm. Simulation results show that LSNC is an efficient network coding scheme for multirate multimedia streaming, and the aggregated number of received layers of both OLSNC and SLSNC is very close to the theoretical upper bound in all configurations analyzed.