The Lip Reading Sentences-3 (LRS3) benchmark has primarily been the focus of intense research in visual speech recognition (VSR) during the last few years. As a result, there is an increased risk of overfitting to its excessively used test set, which is only one hour duration. To alleviate this issue, we build a new VSR test set named WildVSR, by closely following the LRS3 dataset creation processes. We then evaluate and analyse the extent to which the current VSR models generalize to the new test data. We evaluate a broad range of publicly available VSR models and find significant drops in performance on our test set, compared to their corresponding LRS3 results. Our results suggest that the increase in word error rates is caused by the models’ inability to generalize to slightly "harder" and in the wild lip sequences than those found in the LRS3 test set. Our new test benchmark is made public in order to enable future research towards more robust VSR models.

In this work, we study the problem of semantic communication and inference, in which a student agent (i.e. mobile device) queries a teacher agent (i.e. cloud sever) to generate higher-order data semantics living in a simplicial complex. Specifically, the teacher first maps its data into a k-order simplicial complex and learns its high-order correlations. For effective communication and inference, the teacher seeks minimally sufficient and invariant semantic structures prior to conveying information. These minimal simplicial structures are found via judiciously removing simplices selected by the Hodge Laplacians without compromising the inference query accuracy. Subsequently, the student locally runs its own set of queries based on a masked simplicial convolutional autoencoder (SCAE) leveraging both local and remote teacher's knowledge. Numerical results corroborate the effectiveness of the proposed approach in terms of improving inference query accuracy under different channel conditions and simplicial structures. Experiments on a coauthorship dataset show that removing simplices by ranking the Laplacian values yields a 85% reduction in payload size without sacrificing accuracy. Joint semantic communication and inference by masked SCAE improves query accuracy by 25% compared to local student based query and 15% compared to remote teacher based query. Finally, incorporating channel semantics is shown to effectively improve inference accuracy, notably at low signal-to-noise ratio (SNR) values.

Visual Speech Recognition (VSR) differs from the common perception tasks as it requires deeper reasoning over the video sequence, even by human experts. Despite the recent advances in VSR, current approaches rely on labeled data to fully train or finetune their models predicting the target speech. This hinders their ability to generalize well beyond the training set and leads to performance degeneration under out-of-distribution challenging scenarios. Unlike previous works that involve auxiliary losses or complex training procedures and architectures, we propose a simple approach, named Lip2Vec that is based on learning a prior model. Given a robust visual speech encoder, this network maps the encoded latent representations of the lip sequence to their corresponding latents from the audio pair, which are sufficiently invariant for effective text decoding. The generated audio representation is then decoded to text using an off-the-shelf Audio Speech Recognition (ASR) model. The proposed model compares favorably with fully-supervised learning methods on the LRS3 dataset achieving 26 WER. Unlike SoTA approaches, our model keeps a reasonable performance on the VoxCeleb2-en test set. We believe that reprogramming the VSR as an ASR task narrows the performance gap between the two and paves the way for more flexible formulations of lip reading.

Accurate and robust coarse semantic segmentation plays a key role in the pursuit of autonomous driving. We present an algorithm that regularizes the representation space of Semantic Segmentation by Multiple Granularity Learning (SSMGL). This approach explores multiple levels of semantic knowledge in an unified framework, where the fine-grained semantic information can be either labeled or unlabeled. In our experiments, we find that SSMGL can achieve better results (1) on both on-road and off-road benchmarks, (2) under different segmentation architectures, or (3) with different backbones. The method is plug-and-play, not specialized for autonomous driving applications, and can be easily extended to any other segmentation scenario. Moreover, our SSMGL approach does not increase the computational overhead in the inference stage.

Large language models are commonly trained on a mixture of filtered web data and curated ``high-quality'' corpora, such as social media conversations, books, or technical papers. This curation process is believed to be necessary to produce performant models with broad zero-shot generalization abilities. However, as larger models requiring pretraining on trillions of tokens are considered, it is unclear how scalable is curation, and whether we will run out of unique high-quality data soon. At variance with previous beliefs, we show that properly filtered and deduplicated web data alone can lead to powerful models; even significantly outperforming models trained on The Pile. Despite extensive filtering, the high-quality data we extract from the web is still plentiful, and we are able to obtain five trillion tokens from CommonCrawl. We publicly release an extract of 500 billion tokens from our RefinedWeb dataset, and 1.3/7.5B parameters language models trained on it.

Indoor localization has received significant attention over the last decade. For WLANs, Received Signal Strength (RSS) based techniques have been the most popular due to the simplicity of measuring the power and the ubiquity of WiFi infrastructure and devices in different indoor environments. Recently, Time-of-arrival (TOA) techniques has been proposed to be incorporated into IEEE 802.11 standards that promise accurate range estimation via hardware time-stamping. Non-line-of sight (NLOS) propagation is a major challenge and many mitigation algorithms have been proposed but their performance is dependent on the accuracy of NLOS identification. By analyzing the moments (e.g. RMS delay, kurtosis) of the time-domain channel impulse response (CIR), it is possible to infer the condition of the channel. Although these metrics have been verified experimentally for ultra-wideband (UWB) systems, their performance for lower-system bandwidths (e.g. WLAN) has not been addressed in the literature. In this paper, using frequency domain channel measurements in a typical indoor office environment, the accuracy of NLOS identification metrics for WLAN systems is analyzed. It is shown that the accuracy of the RMS delay spread and kurtosis is significantly diminished due to low-time resolution of the CIR. Furthermore, it is also shown that by estimating the coherence bandwidth from the channel transfer function, a better NLOS identification metric is obtained in OFDM-based systems. To the best of the authors' knowledge, this experimental analysis to evaluate NLOS identification metrics has never been reported before.