Given a sample of size $N$, it is often useful to select a subsample of smaller size $n<N$ to be used for statistical estimation or learning. Such a data selection step is useful to reduce the requirements of data labeling and the computational complexity of learning. We assume to be given $N$ unlabeled samples $x_{i}$, and to be given access to a 'surrogate model' that can predict labels $y_i$ better than random guessing. Our goal is to select a subset of the samples, to be denoted by {$x_{i}$}$_{i\in G}$, of size $|G|=n<N$. We then acquire labels for this set and we use them to train a model via regularized empirical risk minimization. By using a mixture of numerical experiments on real and synthetic data, and mathematical derivations under low- and high- dimensional asymptotics, we show that: $(i)$ Data selection can be very effective, in particular beating training on the full sample in some cases; $(ii)$ Certain popular choices in data selection methods (e.g. unbiased reweighted subsampling, or influence function-based subsampling) can be substantially suboptimal.

Video conferencing tools have seen a significant increase in usage in the past few years but they still consume a significant bandwidth of $\sim 100$ Kbps to a few Mbps. In this work, we present Txt2Vid-Web: a practical, web-based, low bandwidth, video conferencing platform building upon the Txt2Vid work [1]. We introduce multiple improvements over the existing Txt2Vid framework – implementing it on browser application stack making it much more accessible and portable, reducing the implementation complexity of the plat-form via WebGL, and implementing a new WebGL shader for ConvTranspose in ONNX runtime – thereby enabling it to run on web-browsers of modern laptops (Fig. 1a). We use WebRTC to establish peer-to-peer data channels over network connections and utilize SDP’s multimedia negotiation scheme over SRTP connections, allowing our platform to provide high-quality video calls for the majority of connections and fall back to Txt2Vid when bandwidth limitations overconstrain SDP’s chosen codecs. We verified our plat-form via subjective study $(n =126)$ consisting of comparison of five different audio-video (AV) contents compressed via standard codecs and Txt2Vid-Web. We choose bitrates of {6 kbps, 10 kbps} for encoding the audio and bitrates of {15 kbps, 35 kbps, 100 kbps} for encoding the video using standard AV codec. Results show that at similar quality of experience our platform requires $100 - 500 \times$ less bandwidth than H.264 and VP9 as video codec and OPUS as audio codec (Fig. 1b). We envision our platform can open up many novel applications. Towards this end, we also open-source both our new tool as a Github repository (https://github.com/tpulkit/txt2vid_browser), and our subjective study dataset (https://tinyurl.com/SubjectiveStudyDataset).

Video represents the majority of internet traffic today, driving a continual race between the generation of higher quality content, transmission of larger file sizes, and the development of network infrastructure. In addition, the recent COVID-19 pandemic fueled a surge in the use of video conferencing tools. Since videos take up considerable bandwidth ($\sim 100$Kbps to a few Mbps), improved video compression can have a substantial impact on network performance for live and pre-recorded content, providing broader access to multimedia content worldwide. We present a novel video compression pipeline, called Txt2Vid, which dramatically reduces data transmission rates by compressing webcam videos (“talking-head videos”) to a text transcript. The text is transmitted and decoded into a realistic reconstruction of the original video using recent advances in deep learning based voice cloning and lip syncing models. Our generative pipeline achieves two to three orders of magnitude reduction in the bitrate as compared to the standard audio-video codecs (encoders-decoders), while maintaining equivalent Quality-of-Experience based on a subjective evaluation by users ($n=242$) in an online study. The Txt2Vid framework opens up the potential for creating novel applications such as enabling audio-video communication during poor internet connectivity, or in remote terrains with limited bandwidth. The code for this work is available athttps://github.com/tpulkit/txt2vid.git.

Human perception is at the core of lossy video compression and yet, it is challenging to collect data that is sufficiently dense to drive compression. In perceptual quality assessment, human feedback is typically collected as a single scalar quality score indicating preference of one distorted video over another. In reality, some videos may be better in some parts but not in others. We propose an approach for collecting finer-grained user feedback through an interactive tool that allows direct optimization of perceptual quality given a fixed bitrate. To this end, we built a novel web-tool which allows users to paint spatio-temporal importance maps over videos. The tool allows for interactive successive refinement: we iteratively re-encode the original video according to the painted importance maps, while maintaining the same bitrate, thus allowing the user to visually see the trade-off of assigning higher importance to one spatio-temporal part of the video at the cost of others. We use this tool to collect data in-the-wild (10 videos, 17 users) and utilize the obtained importance maps in the context of x264 coding to demonstrate that the tool can indeed be used to generate videos which, at the same bitrate, look perceptually better through a subjective study (n = 26) - and are 1.9 times more likely to be preferred by viewers. We plan on collecting a large-scale dataset using the tool for automated perceptual compression in the future. The code for the tool and dataset can be found at https://github.com/jenyap/video-annotation-tool.git.

Banding artifacts are artificially-introduced contours arising from the quantization of a smooth region in a video. Despite the advent of recent higher quality video systems with more efficient codecs, these artifacts remain conspicuous, especially on larger displays. In this work, a comprehensive subjective study is performed to understand the dependence of the banding visibility on encoding parameters and dithering. We subsequently develop a simple and intuitive no-reference banding index called CAMBI (Contrast-aware Multiscale Banding Index) which uses insights from Contrast Sensitivity Function in the Human Visual System to predict banding visibility. CAMBI correlates well with subjective perception of banding while using only a few visually-motivated hyperparameters.

This paper describes an architecture for the massively parallel digitization of neural action potentials. The scheme achieves simultaneous data compression and channel multiplexing through wired-OR interactions within an array of single-slope A/D converters. The achieved compression is lossy but effective at retaining the critical samples belonging to action potential spikes. Simulation results using ex-vivo experimental data from a 512-channel array show compression rates up to ~73x while maintaining ≥90% reconstruction coverage for parasol cells in the primate retina.

We demonstrate a spiking neural circuit for azimuth angle detection inspired by the echolocation circuits of the Horseshoe bat Rhinolophus ferrumequinum and utilize it to devise a model for navigation and target tracking, capturing several key aspects of information transmission in biology. Our network, using only a simple local-information based sensor implementing the cardioid angular gain function, operates at biological spike rate of 10 Hz. The network tracks large angular targets (60 degrees) within 1 sec with a 10% RMS error. We study the navigational ability of our model for foraging and target localization tasks in a forest of obstacles and show that our network requires less than 200X spike-triggered decisions, while suffering only a 1% loss in performance compared to a proportional-integral-derivative controller, in the presence of 50% additive noise. Superior performance can be obtained at a higher average spike rate of 100 Hz and 1000 Hz, but even the accelerated networks requires 20X and 10X lesser decisions respectively, demonstrating the superior computational efficiency of bio-inspired information processing systems.