Deep neural networks have significantly advanced video denoising. However, deploying accurate denoisers across platforms remains challenging, particularly under real-time constraints and with high-resolution inputs. While explicit motion estimation modules improve temporal alignment and enhance denoising quality, they are often omitted in lightweight models due to their computational burden. This paper investigates the trade-off between denoising quality and computational efficiency when using spatially coarse optical flow, and shows that reduced-resolution flow retains most temporal denoising benefits while significantly lowering computational cost. In addition, we introduce an inference-free imbalanced noise pre-training strategy that encourages effective temporal aggregation and complements coarse alignment without modifying the runtime architecture. The proposedLightweightVideoDenoiser (LVD) achieves state-of-the-art performance among lightweight models on RAW video denoising benchmarks while maintaining low latency suitable for real-time edge deployment. Our code is available athttps://github.com/camalab-ai/LVD

We introduce Frame Interpolation Pre-training (FIP), a simple learning technique for lifting deep image denoisers to video denoising with improved implicit temporal alignment. Modern video denoising networks typically rely on explicit motion estimation and alignment which are computationally intensive and harder to re-design and re-train, restricting their application scope and usability. Conversely, stacking frames and image denoisers, without incorporating explicit motion estimation modules, improves speed and benefits from a simpler design, thereby facilitating their generalizability to the video domain. However, it leads to lower accuracy due to suboptimal capture of temporal dependencies. To better leverage the adjacent frames in this setting and reduce the accuracy gap, we propose a novel training regime that divides the standard supervised training of the denoising task into two phases. In the initial phase, FIP guides the network to interpolate a fully masked central frame using only adjacent noisy input frames. In the subsequent phase, the pre-trained network is fine-tuned on denoising the central frame, now using all noisy input frames. Extensive diagnostics indicate that FIP-based networks provide better implicit motion estimation and temporal alignment. In effect, qualitative and quantitative evaluation on standard video denoising datasets with synthetic and real noise demonstrates that FIP consistently improves video denoising accuracy of motion-aware, video-lifted image denoisers without additional computational overhead during training and test time. Our code is available at https://github.com/camalab-ai/FIP.

Optical flow is integral to numerous video processing tasks such as stabilization. Although recent advancements in optical flow estimation have shown significant efficacy in general scenes, their applicability to challenging medical scenarios, which exhibit unique domain-specific visual phenomena, remains limited. Supervised learning methods facilitate the robust training of motion estimators. However, the absence of ground truth optical flow in many medical video-assisted applications poses a significant barrier to their progress. This is particularly evident in Video-Assisted Dentistry (VAD), where enhanced and continual vision could improve the educational, training, and fully operational dental workflows. Addressing this gap, we propose a new method for optical flow adaptation on real videos aided by a large dataset of synthetic dental videos with corresponding ground truth optical flows. Notably, the method employs a novel no-reference evaluation measure for unsupervised learning and incrementally fine-tunes optical flow neural networks. It facilitates their transition from synthetic to real dynamic dental scenes characterized by variable illumination, specular reflections, fluid dynamics, and sparse textures. In effect, this study demonstrates the effective streamed adaptation of multiple state-of-the-art optical flow networks to real medical scenes of video-assisted conservative dental treatments. Our code is available at https://github.com/camalab-ai/sofa-flow.

Infertility affects around 15% of couples worldwide. Male fertility problems include poor sperm quality and low sperm count. The advanced fertility treatment methods like ICSI are nowadays supported by vision systems to assist embryologists in selecting good quality sperm. Computer-Assisted Semen Analysis (CASA) provides quantitative and qualitative sperm analysis concerning concentration, motility, morphology, vitality, and fragmentation. However, fertility assessment algorithms often neglect individual spermatozoon tail and its beating patterns because recognizing the tails in blurry microscopic images reliably is challenging. In this article, we propose that models trained with head and tail part classes can better localize parts and segment the whole spermatozoon objects. Usually, the training of segmentation sperm models is supported by image-level augmentation. We argue that models guided by class-specific data augmentation attend to less discriminative sperm parts. To demonstrate this, we decouple the augmentation into object-level and background augmentation for the sperm part segmentation problem. Our proposed method outperforms state-of-the-art methods on the SegSperm dataset. Moreover, our ablation studies confirm the effectiveness of the proposed part-based object representation and augmentation.

Denoising videos in real-time is critical in many applications, including robotics and medicine, where varying-light conditions, miniaturized sensors, and optics can substantially compromise image quality. This work proposes the first video denoising method based on a deep neural network that achieves state-of-the-art performance on dynamic scenes while running in real-time on VGA video resolution with no frame latency. The backbone of our method is a novel, remarkably simple, temporal network of cascaded blocks with forward block output propagation. We train our architecture with short, long, and global residual connections by minimizing the restoration loss of pairs of frames, leading to a more effective training across noise levels. It is robust to heavy noise following Poisson-Gaussian noise statistics. The algorithm is evaluated on RAW and RGB data. We propose a denoising algorithm that requires no future frames to denoise a current frame, reducing its latency considerably. The visual and quantitative results show that our algorithm achieves state-of-the-art performance among efficient algorithms, achieving from two-fold to two-orders-of-magnitude speed-ups on standard benchmarks for video denoising.

In dentistry, microscopes have become indispensable optical devices for high-quality treatment and micro-invasive surgery, especially in the field of endodontics. Recent machine vision advances enable more advanced, real-time applications including but not limited to dental video deblurring and workflow analysis through relevant metadata obtained by instrument motion trajectories. To this end, the proposed work addresses dental video deblurring and instrument segmentation in a Multi-task Learning fashion, leveraging spatio-temporal adaptive kernels via a recurrent design. The task-specific branches of our architecture employ the responses of those kernels to recover sharper video frames and yield the dental instrument segmentation mask. We demonstrate that the proposed method improves deblurring while retaining segmentation performance under a low computational footprint.

Elongated objects have various shapes and can shift, rotate, change scale, and be rigid or deform by flexing, articulating, and vibrating, with examples as varied as a glass bottle, a robotic arm, a surgical suture, a finger pair, a tram, and a guitar string. This generally makes tracking of poses of elongated objects very challenging. We describe a unified, configurable framework for tracking the pose of elongated objects, which move in the image plane and extend over the image region. Our method strives for simplicity, versatility, and efficiency. The object is decomposed into a chained assembly of segments of multiple parts that are arranged under a hierarchy of tailored spatio-temporal constraints. In this hierarchy, segments can rescale independently while their elasticity is controlled with global orientations and local distances. While the trend in tracking is to design complex, structure-free algorithms that update object appearance on-line, we show that our tracker, with the novel but remarkably simple, structured organization of parts with constant appearance, reaches or improves state-of-the-art performance. Most importantly, our model can be easily configured to track exact pose of arbitrary, elongated objects in the image plane. The tracker can run up to 100 fps on a desktop PC, yet the computation time scales linearly with the number of object parts. To our knowledge, this is the first approach to generic tracking of elongated objects.

Haar-like features are ubiquitous in computer vision, e.g. for Viola and Jones face detection or local descriptors such as Speeded-Up-Robust-Features. They are classically computed in one pass over integral image by reading the values at the feature corners. Here we present a new, general parsing formalism for convolving them more efficiently. Our method is fully automatic and applicable to an arbitrary set of Haar-like features. The parser reduces the number of memory accesses which are the main computational bottleneck during convolution on modern computer architectures. It first splits the features into simpler kernels. Then it aligns and reuses them where applicable forming an ensemble of recursive convolution trees, which can be computed faster. This is illustrated with experiments, which show a significant speed-up over the classic approach.

We propose a novel probabilistic tracking algorithm based on an elastic blob ensemble (EBE) which is applicable to track flexible objects. It outputs a coarse motion cue in the form of the object's location and orientation together with the location of the blobs. The main assumption is that the orientation of the whole object does not change much between neighboring frames. A discrete solution space is created in the current frame around the blobs' positions from the previous frame. Our model then promotes solutions whose orientations are close to the prior orientation, which match the modeled to the observed appearance well, and which follow modeled spatial configuration. It combines the strengths of three popular approaches to visual tracking : mean-shift tracker, particle filtering, and pictorial structures. As a result, the proposed framework tracks human upper body composed of head, torso, and hips at the rate of 40 fps on a regular desktop PC.