Internet of Things (IoT) in extreme environments inevitably produces incomplete multi-view data, presenting challenges to the existing data analysis methods. Although incomplete multi-view clustering methods have the potential to mine patterns of incomplete IoT data, they are still confronted with two challenges. 1) They ignore shifts of semantics caused by missing data in aggregating consistent and complementary information of incomplete data, degrading the robustness of models in pattern mining. 2) Most of them rely on the instances with complete views as pairwise supervision to capture correlations among views, failing to mine inherent patterns of data in the extreme view missing scenario where multi-view instances are only with an available view. To this end, a deep incomplete multi-view clustering network (DIMC) is proposed via defining dual consistencies within the information bottleneck framework to mine accurate patterns of incomplete data. Specifically, an unsupervised multi-view information bottleneck (MIB) is formulated to model dependencies of data, which remedies shifts of semantics via within-view intrinsic knowledge learning, consistent semantics sharing, and consistent structure aligning. Meanwhile, dual consistencies are designed to implement MIB, which builds invariant transformations to mine correlations between views without the help of complete instances. Finally, extensive experiments on four benchmark incomplete datasets demonstrate the superiority of DIMC. Especially, DIMC surpasses the state-of-the-art methods by 0.2048 in accuracy under extreme view missing scenarios.

Multimodal hateful social media meme detection is an important and challenging problem in the vision-language domain. Recent studies show high accuracy for such multimodal tasks due to datasets that provide better joint multimodal embedding to narrow the semantic gap. Religiously hateful meme detection is not extensively explored among published datasets. While there is a need for higher accuracy on religiously hateful memes, deep learning–based models often suffer from inductive bias. This issue is addressed in this work with the following contributions. First, a religiously hateful memes dataset is created and published publicly to advance hateful religious memes detection research. Over 2000 meme images are collected with their corresponding text. The proposed approach compares and fine-tunes VisualBERT pre-trained on the Conceptual Caption (CC) dataset for the downstream classification task. We also extend the dataset with the Facebook hateful memes dataset. We extract visual features using ResNeXT-152 Aggregated Residual Transformations–based Masked Regions with Convolutional Neural Networks (R-CNN) and Bidirectional Encoder Representations from Transformers (BERT) uncased for textual encoding for the early fusion model. We use the primary evaluation metric of an Area Under the Operator Characters Curve (AUROC) to measure model separability. Results show that the proposed approach has a higher AUROC score of 78%, proving the model’s higher separability performance and an accuracy of 70%. It shows comparatively superior performance considering dataset size and against ensemble-based machine learning approaches.

Recent advances in artificial intelligence have led to deepfake images, enabling users to replace a real face with a genuine one. deepfake images have recently been used to malign public figures, politicians, and even average citizens. deepfake but realistic images have been used to stir political dissatisfaction, blackmail, propagate false news, and even carry out bogus terrorist attacks. Thus, identifying real images from fakes has got more challenging. To avoid these issues, this study employs transfer learning and data augmentation technique to classify deepfake images. For experimentation, 190,335 RGB-resolution deepfake and real images and image augmentation methods are used to prepare the dataset. The experiments use the deep learning models: convolutional neural network (CNN), Inception V3, visual geometry group (VGG19), and VGG16 with a transfer learning approach. Essential evaluation metrics (accuracy, precision, recall, F1-score, confusion matrix, and AUC-ROC curve score) are used to test the efficacy of the proposed approach. Results revealed that the proposed approach achieves an accuracy, recall, F1-score and AUC-ROC score of 90% and 91% precision, with our fine-tuned VGG16 model outperforming other DL models in recognizing real and deepfakes.

The volume of multimedia data produced by various smart devices has increased dramatically with the advent of new digital technologies, including in the Internet of Vehicles (IoV). It has become more difficult to extract valuable insights from multimedia data due to several challenges during data analysis. The main problem is the need to quickly and precisely identify abnormalities in multimedia data. This research presents an unusual occurrence of the audio forensics database named UOAFDB and a practical method for identifying and categorizing unusual occurrences in audio files. To study the detection of abnormal audio and the classification of rare sound (e.g., car crash—machine gun, explosion) events for audio forensics, we construct a large audio dataset containing ten rare special events (anomalies) with 15 different background environmental settings (e.g., beach, restaurant, and train). The suggested method determines the optimal amount of features using the best feature extraction methodology available by extracting Mel-frequency cepstral coefficients (MFCCs) features from the audio signals of the newly formed dataset. Modern deep learning algorithms use these features as input to assess performance. Additionally, we apply deep learning methods to the most recent and best available dataset and obtain promising outcomes. The experimental findings demonstrate promising results on the UOAFDB dataset.

The proportion of COVID-19 patients is significantly expanding around the world. Treatment with serious consideration has become a significant problem. Identifying clinical indicators of succession towards severe conditions is desperately required to empower hazard stratification and optimise resource allocation in the pandemic of COVID-19. Consequently, the classification of severity level is significant for the patient’s triaging. It is required to categorise the severity level as mild, moderate, severe, and critical based on the patients’ symptoms. Various symptomatic parameters may encourage the evaluation of infection seriousness. Likewise, with the rapid spread and transmissibility of COVID-19 patients, it is crucial to utilise telemonitoring schemes for COVID-19 patients. Telemonitoring mediation encourages remote data and information exchange among medicinal services, suppliers, and patients, furthermore, risk mitigation and provision of appropriate medical facilities. This paper provides explorative data analysis of symptoms, comorbidities, and other parameters, comparing different machine learning algorithms for case severity detection. This paper also provides a system (based on the degree of truthfulness) for case severity detection that might be utilised to stratify risk levels for anticipated moderate and severe COVID-19 patients. Finally, we provide a telemonitoring model of COVID-19 patients to ensure the remote and continuous monitoring of case severity progression and appropriate risk mitigation strategies.

Detection of human relationships from their interactions on social media is a challenging problem with a wide range of applications in different areas, like targeted marketing, cyber-crime, fraud, defense, planning, and human resource, to name a few. All previous work in this area has only dealt with the most basic types of relationships. The proposed approach goes beyond the previous work to efficiently handle the hierarchy of social relationships. This article introduces a novel technique named Quantifiable Social Relationship (QSR) analysis for quantifying social relationships to analyze relationships between agents from their textual conversations. QSR uses cross-disciplinary techniques from computational linguistics and cognitive psychology to identify relationships. QSR utilizes sentiment and behavioral styles displayed in the conversations for mapping them onto level II relationship categories. Then, for identifying the level III relationship categories, QSR uses level II relationships, sentiments, interactions, and word embeddings as key features. QSR employs natural language processing techniques for feature engineering and state-of-the-art embeddings generated by word2vec, global vectors (glove), and bidirectional encoder representations from transformers (bert). QSR combines the intrinsic conversational features with word embeddings for classifying relationships. QSR achieves an accuracy of up to 89% for classifying relationship subtypes. The evaluation shows that QSR can accurately identify the hierarchical relationships between agents by extracting intrinsic and extrinsic features from textual conversations between agents.

Emotion detection (ED) plays a vital role in determining individual interest in any field. Humans use gestures, facial expressions, and voice pitch and choose words to describe their emotions. Significant work has been done to detect emotions from the textual data in English, French, Chinese, and other high-resource languages. However, emotion classification has not been well studied in low-resource languages (i.e., Urdu) due to the lack of labeled corpora. This article presents a publicly available Urdu Nastalique Emotions Dataset (UNED) of sentences and paragraphs annotated with different emotions and proposes a deep learning (DL)-based technique for classifying emotions in theUNEDcorpus. Our annotatedUNEDcorpus has six emotions for both paragraphs and sentences. We perform extensive experimentation to evaluate the quality of the corpus and further classify it using machine learning and DL approaches. Experimental results show that the developed DL-based model performs better than generic machine learning approaches with an F1 score of 85% on the UNED sentence-based corpus and 50% on the UNED paragraph-based corpus.

Nowadays, much research leverages the clustering to mine commercial patterns from data in enterprise systems. However, previous methods cannot fully consider local structures and global topology of data, which may cause the degradation of clustering performance. To address the challenges, a deep multimodal adversarial cycle-consistent network (DMACCN) is proposed to mine intrinsic patterns of data, which can capture the local structures from instance reconstructions and the global topology from adversarial games. Specifically, DMACCN is designed as an adversarial encoding-decoding architecture composed of the modality specific-encoder, the modality-common fusion network, the cycle-consistent modality-specific generator, and the modality-fusion discriminator, which can fully fuse complementary information of data. Then, an adversarial cycle-consistent loss is devised to guide the clustering pattern mining from complementary information of data, which can align semantics between modalities and capture clustering structures of instances. The two components collaborate in a seamless manner to capture accurate commercial patterns. Finally, extensive experimental results on four datasets show DMACCN greatly outperforms the comparison methods.

Federated learning (FL) has recently emerged as a striking framework for allowing machine and deep learning models with thousands of participants to have distributed training to preserve the privacy of users' data. Federated learning comes with the pros of allowing all participants the possibility of creating robust models even in the absence of sufficient training data. Recently, smartphone usage has increased significantly due to its portability and ability to perform many daily life tasks. Typing on a smartphone's soft keyboard generates vibrations that could be abused to detect the typed keys, aiding side-channel attacks. Such data can be collected using smartphone hardware sensors during the entry of sensitive information such as clinical notes, personal medical information, username, and passwords. This study proposes a novel framework based on federated learning for side-channel attack detection to secure this information. We collected a dataset from 10 Android smartphone users who were asked to type on the smartphone soft keyboard. We convert this dataset into two windows of five users to make two clients training local models. The federated learning-based framework aggregates model updates contributed by two clients and trained the Deep Neural Network (DNN) model individually on the dataset. To reduce the over-fitting factor, each client examines the findings three times. Experiments reveal that the DNN model achieves an accuracy of 80.09%, showing that the proposed framework has the potential to detect side-channel attacks.

Federated learning can effectively ensure data security and improve the problem of data islanding. However, the performance of federated learning-based schemes could be better due to the imbalance of image data. Therefore, this paper proposes a federated learning approach based on priori knowledge and a bilateral segmentation network for image edge extraction. First, federated learning can distribute training images for some special complex images due to the small sample and unshared data. Then, the image with similar edge information to the original image is learned to obtain prior knowledge, and the local uniform sparsity method is used to strengthen the detail features and weaken the background features. Based on the bilateral segmentation network, we introduce a dilated pyramid pooling layer and multi-scale feature fusion module to fuse the shallow detailed features in the context path with the deep abstract features obtained through the dilated pyramid pooling. The final result is obtained by fusing the result with prior knowledge and the result with the context path. Finally, we conduct experiments on some public datasets, and the results show that the proposed method greatly improves extraction accuracy compared with the traditional and the most advanced methods.

With the explosive advancements in smartphone technology, video uploading/downloading has become a routine part of digital social networking. Video contents contain valuable information as more incidents are being recorded now than ever before. In this paper, we present a comprehensive survey on information extraction from video contents and forgery detection. In this context, we review various modern techniques such as computer vision and different machine learning (ML) algorithms including deep learning (DL) proposed for video forgery detection. Furthermore, we discuss the persistent general, resource, legal, and technical challenges, as well as challenges in using DL for the problem at hand, such as the theory behind DL, CV, limited datasets, real-time processing, and the challenges with the emergence of ML techniques used with the Internet of Things (IoT)-based heterogeneous devices. Moreover, this survey presents prominent video analysis products used for video forensics investigation and analysis. In summary, this survey provides a detailed and broader investigation about information extraction and forgery detection in video contents under one umbrella, which was not presented yet to the best of our knowledge.

Summary Smartphones are equipped with precise hardware sensors including accelerometer, gyroscope, and magnetometer. These devices provide real‐time semantic data that can be used to recognize daily life physical activities for personalized smart health assessment. Existing studies focus on the recognition of simple physical activities but they lacked in providing accurate recognition of physical activities having complex interclass variations. Therefore, this research focuses on the accurate recognition of physical activities having complex interclass variations. We propose a two‐layered approach calledPARCIVthat first clusters similar activities based on semantic data and then recognize them using a machine learning classifier. Our two‐layered approach first bounds the highly indistinguishable activities in clusters to avoid misclassification with other distinguishable activities and thereafter recognize them on a fine‐grained level within each cluster. To evaluate our approach, we make an android application that collects labeled data by using smartphone sensors from 10 participants, while performing activities.PARCIVrecognizes distinguishable as well as indistinguishable activities with high accuracy of 99% on the self‐collected dataset. Furthermore,PARCIVachieve 95% accuracy on the publicly available dataset used by state‐of‐the‐art studies.PARCIVoutperforms various state‐of‐the‐art studies by 8%‐17% for simple activities as well as complex activities.

With the recent advancement of smartphone technology in the past few years, smartphone usage has increased on a tremendous scale due to its portability and ability to perform many daily life tasks. As a result, smartphones have become one of the most valuable targets for hackers to perform cyberattacks, since the smartphone can contain individuals’ sensitive data. Smartphones are embedded with highly accurate sensors. This article proposes BetaLogger , an Android-based application that highlights the issue of leaking smartphone users’ privacy using smartphone hardware sensors (accelerometer, magnetometer, and gyroscope). BetaLogger efficiently infers the typed text (long or short) on a smartphone keyboard using Language Modeling and a Dense Multi-layer Neural Network (DMNN). BetaLogger is composed of two major phases: In the first phase, Text Inference Vector is given as input to the DMNN model to predict the target labels comprising the alphabet, and in the second phase, sequence generator module generate the output sequence in the shape of a continuous sentence. The outcomes demonstrate that BetaLogger generates highly accurate short and long sentences, and it effectively enhances the inference rate in comparison with conventional machine learning algorithms and state-of-the-art studies.

Connected and Automated Vehicles (CAVs), owing to their characteristics such as seamless and real-time transfer of data, are imperative infrastructural advancements to realize the emerging smart world. The sensor-generated data are, however, vulnerable to anomalies caused due to faults, errors, and/or cyberattacks, which may cause accidents resulting in fatal casualties. To help in avoiding such situations by timely detecting anomalies, this study proposes an anomaly detection method that incorporates a combination of a multi-stage attention mechanism with a Long Short-Term Memory (LSTM)-based Convolutional Neural Network (CNN), namely, MSALSTM-CNN. The data streams, in the proposed method, are converted into vectors and then processed for anomaly detection. We also designed a method, namely, weight-adjusted fine-tuned ensemble: WAVED, which works on the principle of average predicted probability of multiple classifiers to detect anomalies in CAVs and benchmark the performance of the MSALSTM-CNN method. The MSALSTM-CNN method effectively enhances the anomaly detection rate in both low and high magnitude cases of anomalous instances in the dataset with the gain of up to 2.54% in F-score for detecting different single anomaly types. The method achieves the gain of up to 3.24% in F-score in the case of detecting mixed anomaly types. The experiment results show that the MSALSTM-CNN method achieves promising performance gain for both single and mixed multi-source anomaly types as compared to the state-of-the-art and benchmark methods.

In this digital age, human dependency on technology in various fields has been increasing tremendously. Torrential amounts of different electronic products are being manufactured daily for everyday use. With this advancement in the world of Internet technology, cybersecurity of software and hardware systems are now prerequisites for major business’ operations. Every technology on the market has multiple vulnerabilities that are exploited by hackers and cyber-criminals daily to manipulate data sometimes for malicious purposes. In any system, the Intrusion Detection System (IDS) is a fundamental component for ensuring the security of devices from digital attacks. Recognition of new developing digital threats is getting harder for existing IDS. Furthermore, advanced frameworks are required for IDS to function both efficiently and effectively. The commonly observed cyber-attacks in the business domain include minor attacks used for stealing private data. This article presents a deep learning methodology for detecting cyber-attacks on the Internet of Things using a Long Short Term Networks classifier. Our extensive experimental testing show an Accuracy of 99.09%, F1-score of 99.46%, and Recall of 99.51%, respectively. A detailed metric representing our results in tabular form was used to compare how our model was better than other state-of-the-art models in detecting cyber-attacks with proficiency.