A significant proportion of the world's population speaks Urdu and Hindi, with many individuals being bilingual in both English and these languages. Still, no multilingual spoofing dataset exists to capture the conversational style of bilingual speakers who frequently code-switch while communicating. This paper presents a multilingual spoofed speech (MSS) dataset comprising 472,486 utterances from 154 speakers. We specifically considered bona fide utterances from Urdu and Hindi speakers, where language alternation occurs within a single audio. Spoofed samples are generated using voice conversion techniques to preserve the speaking accents and conversation styles of bilingual individuals. Further, we propose and evaluate an anti-spoofing framework called WavSpeech-AASIST, which incorporates self-supervised models (wav2vec and UniSpeech) into the AASIST network. Our comparative analysis underscores the significance of the MSS dataset and demonstrates the effectiveness of WavSpeech-AASIST for audio spoofing detection.

The evolution of artificial intelligence (AI) techniques in recent years has increased the generation of fake content including AI-generated text, images, audio, and videos. Among which the fake visual content commonly known as deepfakes has imposed a great threat to society due to its negative impacts. To mitigate the adverse aspects of deepfakes, the research community has introduced various deepfakes detection methods. However, these deepfakes detection methods lack the interpretability and explainability of the decision-making process. The interpretable model increases trustworthiness as it provides the reasoning for classifying outcomes as real or fake. Therefore, in this paper, we have introduced ConvNext-PNet, which is a prototypical-based learning framework for the interpretable and explainable detection of visual deepfakes. In the proposed framework, prototype learning is incorporated into the modified ConvNext model that improves the discriminative features learning capability of the proposed framework along with the explainability aspect. The performance of ConvNext-PNet is evaluated on challenging datasets including FaceForensics++ (FF++), CelebDF, DFDC-P, and DeepFakeFace (DFF) datasets. The robustness of the proposed model is validated through various experiments along with the interpretability analysis. The quantitative results demonstrate the effectiveness of the model for the detection of visual manipulation, whereas the model interpretability and explainability aspect increases the trustworthiness via providing reasoning for the model predictions.

Recently, we have observed an exponential growth in highly realistic deepfake videos, which are often used to spread disinformation, defame individuals, and even influence political outcomes. To combat these manipulated videos, researchers have proposed various deepfake detection techniques. Recent research has revealed that these detection techniques are vulnerable to different adversarial attacks. This paper examines the vulnerability of deepfake detectors to adversarial black-box attacks in terms of performing penetration testing to expose the existing defense benchmarks of current deepfake detectors. We present a perceptual facial mole black-box adversarial attack on deepfake detectors, where the attacker has limited knowledge of the architecture and settings of the detector. The proposed attack is visually natural and transferable based on the attention distraction mechanism, which distracts the model-shared attention patterns from the region of interest to other regions. We illustrate the efficacy of our attack on multiple cutting-edge deepfake detectors. This attack demonstrates that small perceptible perturbations that are visually natural on the facial face can disrupt and reduce the accuracy of the detectors significantly, up to 40.3%, with the highest success rate of 48.7%. Our findings highlight the necessity for proposing effective deepfake detectors that are resistant to black-box attacks.

Abstract There is demand for scalable algorithms capable of clustering and analyzing large time series data. The Kohonen self-organizing map (SOM) is an unsupervised artificial neural network for clustering, visualizing, and reducing the dimensionality of complex data. Like all clustering methods, it requires a measure of similarity between input data (in this work time series). Dynamic time warping (DTW) is one such measure, and a top performer that accommodates distortions when aligning time series. Despite its popularity in clustering, DTW is limited in practice because the runtime complexity is quadratic with the length of the time series. To address this, we present a new a self-organizing map for clustering TIME Series, called SOMTimeS, which uses DTW as the distance measure. The method has similar accuracy compared with other DTW-based clustering algorithms, yet scales better and runs faster. The computational performance stems from the pruning of unnecessary DTW computations during the SOM’s training phase. For comparison, we implement a similar pruning strategy for K-means, and call the latter K-TimeS. SOMTimeS and K-TimeS pruned 43% and 50% of the total DTW computations, respectively. Pruning effectiveness, accuracy, execution time and scalability are evaluated using 112 benchmark time series datasets from the UC Riverside classification archive, and show that for similar accuracy, a 1.8 $$\times$$ × speed-up on average for SOMTimeS and K-TimeS, respectively with that rates vary between 1 $$\times$$ × and 18 $$\times$$ × depending on the dataset. We also apply SOMTimeS to a healthcare study of patient-clinician serious illness conversations to demonstrate the algorithm’s utility with complex, temporally sequenced natural language.

The rise of deepfake videos poses a serious threat to the authenticity of visual media, as they have a potential to manipulate public opinion, mislead individuals or groups, harm reputation, etc. Traditional methods for detecting deepfakes rely on deep learning models, which lack transparency and interpretability. To gain the confidence of forensic experts in AI-based deepfakes detector, we present a novel DFP-Net for detecting deepfakes using interpretable and explainable prototypes. Our method makes use of the power of prototype-based learning to generate a set of representative images that capture the essential features of genuine and deepfake images. These prototypes are then used to explain our model’s decision-making process and to provide insights into the features most relevant for deepfake detection. We then use these prototypes to train a classification model that can detect deepfakes accurately and with high interpretability. To further improve the interpretability of our method, we also utilize the Grad-CAM technique to generate heatmaps that highlight the regions of the image that contribute the most towards the decision of the model. These heatmaps can be used to explain the reasoning behind the model’s decision and provide insights into the visual cues that distinguish deepfakes from real images. Experimental results on a large-scale FaceForensics++, Celeb-DF and DFDC-P datasets demonstrate that our method achieves state-of-the-art performance in deepfakes detection. Moreover, the interpretability and explainability of our method make it more trustworthy to forensic experts by allowing them to understand how the model works and makes predictions.

Automated analysis of sports video summarization is challenging due to variations in cameras, replay speed, illumination conditions, editing effects, game structure, genre, etc. To address these challenges, we propose an effective video summarization framework based on shot classification and replay detection for field sports videos. Accurate shot classification is mandatory to better structure the input video for further processing, i.e., key events or replay detection. Therefore, we present a lightweight convolutional neural network based method for shot classification. Then we analyze each shot for replay detection and specifically detect the successive batch of logo transition frames that identify the replay segments from the sports videos. For this purpose, we propose local octa-pattern features to represent video frames and train the extreme learning machine for classification as replay or non-replay frames. The proposed framework is robust to variations in cameras, replay speed, shot speed, illumination conditions, game structure, sports genre, broadcasters, logo designs and placement, frame transitions, and editing effects. The performance of our framework is evaluated on a dataset containing diverse YouTube sports videos of soccer, baseball, and cricket. Experimental results demonstrate that the proposed framework can reliably be used for shot classification and replay detection to summarize field sports videos.

The growing number of voice-enabled devices and applications consider automatic speaker verification (ASV) a fundamental component. However, maximum outreach for ASV in critical domains e.g., financial services and health care, is not possible unless we overcome security breaches caused by voice cloning algorithms and replayed audios. Therefore, to overcome these vulnerabilities, a secure ASV (SASV) system based on the novel sign modified acoustic local ternary pattern (sm-ALTP) features and asymmetric bagging-based classifier-ensemble with enhanced attack vector is presented. The proposed audio representation approach clusters the high and low frequency components in audio frames by normally distributing frequency components against a convex function. Then, the neighborhood statistics are applied to capture the user specific vocal tract information. The proposed SASV system simultaneously verifies the bonafide speakers and detects the voice cloning attack, cloning algorithm used to synthesize cloned audio (in the defined settings), and voice-replay attacks over the ASVspoof 2019 dataset. In addition, the proposed method detects the voice replay and cloned voice replay attacks over the VSDC dataset. Both the voice cloning algorithm detection and cloned-replay attack detection are novel concepts introduced in this paper. The voice cloning algorithm detection module determines the voice cloning algorithm used to generate the fake audios. Whereas, the cloned voice replay attack detection is performed to determine the SASV behavior when audio samples are simultaneously contemplated with cloning and replay artifacts.

Sports broadcasters generate an enormous amount of video content on the cyberspace due to massive viewership all over the world. Analysis and consumption of this huge repository urges the broadcasters to apply video summarisation to extract the exciting segments from the entire video to capture user's interest and reap the storage and transmission benefits. Therefore, in this study an automatic method for key‐events detection and summarisation based on audio‐visual features is presented for cricket videos. Acoustic local binary pattern features are used to capture excitement level in the audio stream, which is used to train a binary support vector machine (SVM) classifier. Trained SVM classifier is used to label audio frame as an excited or non‐excited frame. Excited audio frames are used to select candidate key‐video frames. A decision tree‐based classifier is trained to detect key‐events in the input cricket videos that are then used for video summarisation. Performance of the proposed framework has been evaluated on a diverse dataset of cricket videos belonging to different tournaments and broadcasters. Experimental results indicate that the proposed method achieves an average accuracy of 95.5%, which signifies its effectiveness.

The elderly people living alone or life of a patient face distress situations particularly in case of falling and becoming unable to ask for help. Fall in elderly people may result in head injury, broken hips, and bones that need immediate hospitalization to lower the mortality risk. During the last decade, several technological solutions were presented for early fall detection but most of them have critical limitations and are impeded by several environmental constraints. In this paper, we have analyzed the environmental sounds for early fall detection utilizing the fact that reflection of pain directly occurs through sound. The proposed framework first analyzes the environmental sounds by suppressing the silence zones in signals and distinguishing overlapping sound signals through hidden Markov model based component analysis (HMM-CA). The source separated components are then represented by acoustic local ternary patterns (acoustic-LTPs) by extending the existing ideas of acoustic local binary patterns (acoustic-LBPs). In the proposed work, we have also introduced the concept of rotation invariance through uniform patterns for audio signals that, arguably, is a fundamental requirement for an acoustic descriptor. Once the signal representation is completed, we classify the signals through SVM classifier. The performance of the proposed acoustic-LTP is evaluated against state-of-the-art methods and acoustic-LBP. Results clearly evince that proposed method is more powerful and reliable in terms of fall detection when compared against other methods.

This letter presents a framework for replay detection in sports videos to generate highlights. For replay detection, the proposed work exploits the following facts: 1) broadcasters introduce gradual transition (GT) effect both at the start and at the end of a replay segment (RS), and 2) the absence of score captions (SCs) in an RS. The dual-threshold-based method is used to detect GT frames from the input video. A pair of successive GT frames is used to extract the candidate RSs. All frames in the selected segment are processed to detect SC. To this end, temporal running average is used to filter out temporal variations. First- and second-order statistics are used to binarize the running average image, which is fed to optical character recognition stage for character recognition. The absence/presence of SC is used for replay/live frame labeling. The SC detection stage complements the GT detection process, therefore, a combination of both is expected to result in superior computational complexity and detection accuracy. The performance of the proposed system is evaluated on 22 videos of four different sports (e.g., Cricket, tennis, baseball, and basketball). Experimental results indicate that the proposed method can achieve average detection accuracy ≥ 94.7%.