Recently the performance of long-term time series forecasting has been greatly improved by deep models. In this paper, we propose a general framework called Multi-scale Moving Transformation (MMT) that can be applied to the state-of-the-art models of time series forecasting tasks. Via comparative experiments, we demonstrate the effectiveness of the MMT framework against the models with the moving average. By incorporating the MMT framework with state-of-the-art backbone models based on different methods, our experimental results on various public datasets demonstrate that the improvements outperform their corresponding baseline counterparts.

Voice pathology detection plays an important role in diagnosis and medical intervention. Existing methods suffer from inferior performance with limited and imbalanced training samples since the pathological information is coupled with other linguistic and paralinguistic attributes. In this work, a novel adversarial feature disentanglement framework is proposed to achieve the voice pathology detection task by extracting task-oriented features and optimizing feature space. Specifically, to suppress noise features in the coupled representations, an adversarial feature disentanglement mechanism is proposed to decouple pathological and non-pathological information, in which a mutual information discriminator is introduced to prevent information leakage. A classification and contrastive learning (CCL) module is designed to cluster intra-class embeddings in high-dimensional space. Experiments on the open-source SVD and FEMH datasets demonstrate that the proposed model outperforms other competitive baselines, achieving 87.94% and 92.06% accuracy, respectively. The visualization also validates the effectiveness in distinguishing pathological and non-pathological features.

Audio-visual speech recognition (AVSR) leverages complementary visual cues to improve speech recognition. However, in real-world scenarios, both modalities may suffer from noise or occlusion. In such scenarios, most existing fusion strategies overlook the variation in modality-specific quality under different degradation conditions. This limitation may lead to dominance of corrupted modality in the fusion process, resulting in worse AVSR performance than unimodal systems, termed as Corrupted Modality Bias (CMB) in this work. To address this, a self-supervised speech representation learning framework, called AV-RISE, is proposed to employ teacher-student self-distillation to robustly reconstruct clean speech representations from corrupted audio-visual inputs. A hierarchical fusion mechanism is designed to progressively refine audio and visual representations by integrating the Suppression and Enhancement Interaction (SEI) module into each layer of the pre-trained encoder. In the SEI module, cross-modal suppression and modality-oriented enhancement are performed to mitigate noise-induced feature inconsistencies, which strengthens the modeling of complementary semantic representations. Extensive experiments on the LRS2 and LRS3 datasets demonstrate that AV-RISE outperforms SOTA AVSR models, especially under extreme degradation. Most importantly, the hierarchical SEI-based fusion effectively enhances reliable semantic representations to mitigate CMB, by evaluating feature similarities between clean and noise samples.

Flight Trajectory Prediction (FTP) is an essential task in Air Traffic Control (ATC), which can assist air traffic controllers in managing airspace more safely and efficiently. Existing methods generally perform multi-horizon FTP tasks in an autoregressive manner, thereby suffering from error accumulation and low-efficiency problems. In this paper, a novel framework, called FlightBERT++, is proposed to i) forecast multi-horizon flight trajectories directly in a non-autoregressive way, and ii) improve the limitation of the binary encoding (BE) representation in the FlightBERT framework. Specifically, the proposed framework is implemented by a generalized encoder-decoder architecture, in which the encoder learns the temporal-spatial patterns from historical observations and the decoder predicts the flight status for the future horizons. Compared to conventional architecture, an innovative horizon-aware context generator is dedicatedly designed to consider the prior horizon information, which further enables non-autoregressive multi-horizon prediction. Additionally, the Gray code representation and the differential prediction paradigm are designed to cope with the high-bit misclassifications of the BE representation, which significantly reduces the outliers in the predictions. Moreover, a differential prompted decoder is proposed to enhance the capability of the differential predictions by leveraging the stationarity of the differential sequence. Extensive experiments are conducted to validate the proposed framework on a real-world flight trajectory dataset. The experimental results demonstrated that the proposed framework outperformed the competitive baselines in both FTP performance and computational efficiency. The code is publicly available at: https://github.com/gdy-scu/FlightBERT_PP_V2

Accurate recognition of named entities from spoken instructions remains a significant challenge for automatic speech recognition (ASR) techniques in air traffic control (ATC), which limits the reliability of ASR-based applications. A promising solution to overcome this challenge is to integrate prior contextual knowledge into ASR since it contains rich named entities used in ATC communications. Although existing studies have investigated ATC-related contextual ASR techniques, there is a lack of benchmarks to evaluate the advantages of different approaches. In this article, a comprehensive comparative study is presented to explore effective contextual ASR approaches for the ATC domain. Specifically, several typical contextual ASR approaches are introduced in ATC to conduct a comprehensive comparison. Moreover, a novel contextual ASR model, denoted CATCNet, is presented to dedicatedly address the domain-specific problems in ATC, such as limited resources, fast speech, and volatile noise. Several evaluation metrics are proposed to validate the performance of comparison approaches based on the practical requirements of ATC efforts. Extensive experiments are conducted across two real-world ATC speech corpora to build the benchmark. The experimental results demonstrated that integrating context knowledge is effective in improving the recognition performance of named entities. Crucially, the proposed CATCNet outperforms other baseline models by confirming all technical improvements, achieving 80.0% and 86.54% instruction recognition accuracy (IRA) on the ATCSpeech and C-ATCSpeech corpora, respectively. It is believed that this work not only overcomes the bottleneck of ASR performance in the ATC domain, but also provides an applicable solution for ATC-related ASR applications.

Flight Trajectory Prediction (FTP) is an essential task in Air Traffic Control (ATC), which can assist air traffic controllers in managing airspace more safely and efficiently. Existing approaches generally perform multi-horizon FTP tasks in an autoregressive manner, thereby suffering from error accumulation and low-efficiency problems. In this paper, a novel framework, called FlightBERT++, is proposed to i) forecast multi-horizon flight trajectories directly in a non-autoregressive way, and ii) improve the limitation of the binary encoding (BE) representation in the FlightBERT. Specifically, the FlightBERT++ is implemented by a generalized encoder-decoder architecture, in which the encoder learns the temporal-spatial patterns from historical observations and the decoder predicts the flight status for the future horizons. Compared with conventional architecture, an innovative horizon-aware contexts generator is dedicatedly designed to consider the prior horizon information, which further enables non-autoregressive multi-horizon prediction. Moreover, a differential prompted decoder is proposed to enhance the capability of the differential predictions by leveraging the stationarity of the differential sequence. The experimental results on a real-world dataset demonstrated that the FlightBERT++ outperformed the competitive baselines in both FTP performance and computational efficiency.

Radio speech echo is a specific phenomenon in the air traffic control (ATC) domain, which degrades speech quality and further impacts automatic speech recognition (ASR) accuracy. In this work, a time-domain recognition-oriented speech enhancement (ROSE) framework is proposed to improve speech intelligibility and also advance ASR accuracy based on convolutional encoder-decoder-based U-Net framework, which serves as a plug-and-play tool in ATC scenarios and does not require additional retraining of the ASR model. Specifically, 1) In the U-Net architecture, an attention-based skip-fusion (ABSF) module is applied to mine shared features from encoders using an attention mask, which enables the model to effectively fuse the hierarchical features. 2) A channel and sequence attention (CSAtt) module is innovatively designed to guide the model to focus on informative features in dual parallel attention paths, aiming to enhance the effective representations and suppress the interference noises. 3) Based on the handcrafted features, ASR-oriented optimization targets are designed to improve recognition performance in the ATC environment by learning robust feature representations. By incorporating both the SE-oriented and ASR-oriented losses, ROSE is implemented in a multi-objective learning manner by optimizing shared representations across the two task objectives. The experimental results show that the ROSE significantly outperforms other state-of-the-art methods for both the SE and ASR tasks, in which all the proposed improvements are confirmed by designed experiments. In addition, the proposed approach can contribute to the desired performance improvements on public datasets.

Air Traffic Control (ATC) is a complicated, time-evolving, and real-time procedure to direct flight operations in a safer and ordered manner. Although enormous data storages are available during air traffic operations for over 40 years, data-driven intelligent application in aviation is still an emerging task due to the safety-critical issue. With the prevalence of the Next Generation ATC system, artificial intelligence (AI) -empowered research topics are attracting increasing attention from both industrial and academic domains and a high-quality dataset naturally becomes the prerequisite for such practices. However, almost all ATC-related datasets are only unimodal for certain tasks, which fails to comprehensively illustrate the traffic situation to further support real-world studies. To address this gap, a multimodal air traffic situation (M2ATS) dataset is constructed to advance AI-related research in the ATC domain, including airspace information, flight plan, trajectory, and speech. M2ATS covers 10362 flights ATC situation data, involving 110000+ utterances (104 hours) with diversity golden text annotations, 16 intents, and 51 slots. Considering the real-world ATC requirements, a total of 10 multimedia-related tasks (24 baselines) are designed to validate the proposed dataset, covering automatic speech recognition, natural language processing, and spatial-temporal data processing. New ATC-related metrics corresponding to ATC applications are proposed in addition to the common metrics to evaluate task performance. Extensive experiment results demonstrate that the selective baselines can achieve designed tasks on this new dataset, and further investigations are also required to address task and data specificities. It is believed that the proposed new dataset is a new practice to advance AI applications to an industrial scene, which not only promotes ATC-related applications but also provides diverse research topics in the common multimedia community.

Automatic spoken instruction understanding (SIU) of the controller-pilot conversations in the air traffic control (ATC) requires not only recognizing the words and semantics of the speech but also determining the role of the speaker. However, few of the published works on the automatic understanding systems in air traffic communication focus on speaker role identification (SRI). In this article, we formulate the SRI task of controller-pilot communication as a binary classification problem. Furthermore, the text-based, speech-based, and speech-and-text-based multi-modal methods are proposed to achieve a comprehensive comparison of the SRI task. To ablate the impacts of the comparative approaches, various advanced neural network architectures are applied to optimize the implementation of text-based and speech-based methods. Most importantly, a multi-modal speaker role identification network (MMSRINet) is designed to achieve the SRI task by considering both the speech and textual modality features. To aggregate modality features, the modal fusion module is proposed to fuse and squeeze acoustic and textual representations by modal attention mechanism and self-attention pooling layer, respectively. Finally, the comparative approaches are validated on the ATCSpeech corpus collected from a real-world ATC environment. The experimental results demonstrate that all the comparative approaches worked for the SRI task, and the proposed MMSRINet shows competitive performance and robustness compared with the other methods on both seen and unseen data, achieving 98.56% and 98.08% accuracy, respectively.

In the air traffic control (ATC) domain, automatic speech recognition (ASR) suffers from radio speech echo, which cannot be addressed by existing echo cancellation due to auditory-oriented optimization and poor generalization ability caused by volatile radio transmission. In this work, a contrastive learning-based framework is proposed to tackle the radio-echo speech for the ASR task based on convolution networks with multiple paths and recurrent neural networks. 1) By analyzing the communication mechanism of the ATC speech, a novel transmission method is designed to collect clean and noisy speech samples (with the same texts) via a bypass device in a real-world ATC environment. 2) To enhance the model capacity, a temporal and frequency attention block is innovatively designed to guide the model to focus on informative frames and frequencies, aiming at learning shared representations between the clean and noisy speech signals with the same texts. 3) By incorporating contrastive loss, the proposed approach is implemented by a multi-objective optimization, in which the loss weights are dynamically determined to enhance the ASR performance in a learnable manner. With the proposed transmission method, a real-world dataset is collected and annotated to validate the proposed approach. Experimental results demonstrate that the proposed approach outperforms other comparative baselines with different technical frameworks, achieving a 6.76% character error rate on the test dataset. Most importantly, all the proposed improvements are confirmed by designed experiments, in which contrastive learning with learnable multi-objective loss weights contributes to the primary performance improvement.

Flight Trajectory Prediction (TP) is an essential task in Air Traffic Control (ATC). Currently, the TP task is usually achieved by regression approaches, which concatenates several scalar attributes of the observation into a low-dimensional vector as the inputs. However, it is difficult to accurately model aircraft motion patterns using low-dimensional features in complex and time-varying ATC environments. To improve the performance of the TP task, in this paper, a novel framework, called FlightBERT, is proposed based on Binary Encoding (BE) representation, which enables us to tackle the TP task as a multi binary classification problem. Specifically, the scalar attributes of the flight trajectory are encoded into binary codes and transformed into a high-dimensional representation by the attribute embedding module. Considering the prior knowledge among flight attributes, an Attribute Correlation Attention (ACoAtt) block is designed to explicitly capture the correlations among the specific attributes. A stacked Transformer block is applied to serve as the backbone network, which is followed by the predictor to generate the outputs. Considering the nature of flight trajectory, a hybrid constrained loss, i.e., combining the mean square error loss with the binary cross-entropy loss, is innovatively designed to optimize the proposed framework. The proposed method is validated on a large-scale dataset, which is collected from the real-world ATC environment. The experimental results demonstrate that the proposed method outperforms other baselines by quantitative and qualitative evaluations.

In this work, a deep learning-based framework is proposed to implement an autonomous pilot agent (APA), which serves as a human pseudo-pilot to assist air traffic controller (ATCO) training. A novel paradigm, including speech recognition, language understanding, pilot repetition generation (PRG), and text-to-speech (TTS), is designed to formulate the framework pipeline, which also incorporates a simulation system interface. We mainly focus on the PRG and TTS models to address the ATC specificities in this work. The neural architecture is proposed to generate the text repetition instruction by using a sequence-to-sequence text mapping. The Transformer block is improved to implement a high-efficient TTS model, in which the nonautoregressive mechanism is applied to achieve the parallel synthesis. A dedicated phoneme vocabulary is designed to cope with the multilingual issue in the ATC domain and address the out-of-vocabulary problem. With the APA framework, a virtual training mode is proposed to complete the training task without the limitation of time and location. Experimental results on a real-world dataset show that the proposed APA framework replaces the human pilot with considerable high confidence in a real-time manner during the simulation training. Most importantly, the APA framework and the virtual training system are able to cope with the dilemma of physical attendance (like COVID-19) and improve the equipment utilization capacity for the ATCO training.

This work focuses on robust speech recognition in air traffic control (ATC) by designing a novel processing paradigm to integrate multilingual speech recognition into a single framework using three cascaded modules: an acoustic model (AM), a pronunciation model (PM), and a language model (LM). The AM converts ATC speech into phoneme-based text sequences that the PM then translates into a word-based sequence, which is the ultimate goal of this research. The LM corrects both phoneme- and word-based errors in the decoding results. The AM, including the convolutional neural network (CNN) and recurrent neural network (RNN), considers the spatial and temporal dependences of the speech features and is trained by the connectionist temporal classification loss. To cope with radio transmission noise and diversity among speakers, a multiscale CNN architecture is proposed to fit the diverse data distributions and improve the performance. Phoneme-to-word translation is addressed via a proposed machine translation PM with an encoder-decoder architecture. RNN-based LMs are trained to consider the code-switching specificity of the ATC speech by building dependences with common words. We validate the proposed approach using large amounts of real Chinese and English ATC recordings and achieve a 3.95% label error rate on Chinese characters and English words, outperforming other popular approaches. The decoding efficiency is also comparable to that of the end-to-end model, and its generalizability is validated on several open corpora, making it suitable for real-time approaches to further support ATC applications, such as ATC prediction and safety checking.

Opportunistic underwater sensor networks (OUSNs) are developed for a set of underwater applications, including underwater creatures tracking and tactical surveillance. However, the storage capacity of nodes is sometimes insufficient, especially compared to a wealth of data messages which are generated rapidly in some emergency response applications. Therefore, the network throughput should be taken as one of the primary objectives of message dissemination. To this end, the strategies for message storing, disseminating and discarding are investigated, and a Message Dissemination Approach for Storage-Limited (MDA-SL) OUSNs is proposed hereby. In MDA-SL, the messages are preferred to be disseminated to the nodes with higher speed or larger residual storage. In addition, the newer messages are inclined to be discarded when their holders' storage is full. Furthermore, through simulation analysis, the performance of MDA-SL is proved excellent, which indicates that MDA-SL achieves a satisfactory throughput with the propagation delay being restricted according to application requirements.