Recent advances in vision backbones have yielded powerful and diverse visual and video encoders. Yet, current Video Large Language Models encode visual inputs using an encoder from a single backbone family, limiting the amount and type of visual information they can process. We propose MERV, a Multi-Encoder Video Representation, which utilizes multiple encoders for a comprehensive video representation. To optimize heterogeneous features from a broad spectrum of encoders and ensure efficient and coherent feature integration, MERV first aligns encoder features spatio-temporally, then projects them into a unified structure, and finally fuses them through cross-attention. Under fair comparison, MERV achieves up to 4.62% higher accuracy than its base model, while introducing minimal extra parameters and training faster than equivalent single-encoder methods after parallelizing visual processing. Qualitative analysis shows MERV successfully captures and integrates domain knowledge from each encoder, opening new possibilities for scaling enhanced video understanding.

Sensor technology developments provide a basis for effective fault diagnosis in manufacturing systems. However, the limited number of sensors due to physical constraints or undue costs hinders the accurate diagnosis in the actual process. In addition, time-varying operational conditions that generate nonstationary process faults and the correlation information in the process require to consider for accurate fault diagnosis in the manufacturing systems. This article proposes a novel fault diagnosis method: clustering spatially correlated sparse Bayesian learning (CSSBL), and explicitly demonstrates its applicability in a multistation assembly system that is vulnerable to the above challenges. Specifically, the method is based on a practical assumption that it will likely have a few process faults (sparse). In addition, the hierarchical structure of CSSBL has several parameterized prior distributions to address the above challenges. As posterior distributions of process faults do not have closed form, this paper derives approximate posterior distributions through Variational Bayes inference. The proposed method’s efficacy is provided through numerical and real-world case studies utilizing an actual autobody assembly system. The generalizability of the proposed method allows the technique to be applied in fault diagnosis in other domains, including communication and healthcare systems. Note to Practitioners—This article proposes a new process fault diagnosis method: clustering spatially correlated sparse Bayesian learning. This method effectively diagnoses time-varying defects by leveraging the correlation structures in the process when sensor measurements are insufficient. The actual autobody assembly process is utilized to show the proposed method’s effectiveness. The proposed method performs superior to the benchmark methods in fault detection capability. In addition, the proposed method accurately estimates the severity of the process faults, providing significant information to the practitioners for their decision-making in the maintenance schedule. Specifically, the error between the estimation from the proposed method and the actual severity of the process faults achieves less than 10% of error of all the benchmark methods when there exists a high correlation between the variations of the fixture locators in the autobody assembly system.

Differential equations are fundamental in modeling numerous physical systems, including thermal, manufacturing, and meteorological systems. Traditionally, numerical methods often approximate the solutions of complex systems modeled by differential equations. With the advent of modern deep learning, Physics-informed Neural Networks (PINNs) are evolving as a new paradigm for solving differential equations with a pseudo-closed form solution. Unlike numerical methods, the PINNs can solve the differential equations mesh-free, integrate the experimental data, and resolve challenging inverse problems. However, one of the limitations of PINNs is the poor training caused by using the activation functions designed typically for purely data-driven problems. This work proposes a scalable tanh-based activation function for PINNs to improve learning the solutions of differential equations. The proposed Self-scalable tanh (Stan) function is smooth, non-saturating, and has a trainable parameter. It can allow an easy flow of gradients and enable systematic scaling of the input-output mapping during training. Various forward problems to solve differential equations and inverse problems to find the parameters of differential equations demonstrate that the Stan activation function can achieve better training and more accurate predictions than the existing activation functions for PINN in the literature.

It is well-known in the video understanding community that human action recognition models suffer from background bias, i.e., over-relying on scene cues in making their predictions. However, it is difficult to quantify this effect using existing evaluation frameworks. We introduce the Human-centric Analysis Toolkit (HAT), which enables evaluation of learned background bias without the need for new manual video annotation. It does so by automatically generating synthetically manipulated videos and leveraging the recent advances in image segmentation and video inpainting. Using HAT we perform an extensive analysis of 74 action recognition models trained on the Kinetics dataset. We confirm that all these models focus more on the scene background than on the human motion; further, we demonstrate that certain model design decisions (such as training with fewer frames per video or using dense as opposed to uniform temporal sampling) appear to worsen the background bias. We open-source HAT to enable the community to design more robust and generalizable human action recognition models.

We contribute HAA5001, a manually annotated human-centric atomic action dataset for action recognition on 500 classes with over 591K labeled frames. To minimize ambiguities in action classification, HAA500 consists of highly diversified classes of fine-grained atomic actions, where only consistent actions fall under the same label, e.g., "Baseball Pitching" vs "Free Throw in Basketball". Thus HAA500 is different from existing atomic action datasets, where coarse-grained atomic actions were labeled with coarse action-verbs such as "Throw". HAA500 has been carefully curated to capture the precise movement of human figures with little class-irrelevant motions or spatiotemporal label noises.The advantages of HAA500 are fourfold: 1) human-centric actions with a high average of 69.7% detectable joints for the relevant human poses; 2) high scalability since adding a new class can be done under 20–60 minutes; 3) curated videos capturing essential elements of an atomic action without irrelevant frames; 4) fine-grained atomic action classes. Our extensive experiments including cross-data validation using datasets collected in the wild demonstrate the clear benefits of human-centric and atomic characteristics of HAA500, which enable training even a baseline deep learning model to improve prediction by attending to atomic human poses. We detail the HAA500 dataset statistics and collection methodology and compare quantitatively with existing action recognition datasets.

State-of-the-art semantic segmentation methods were almost exclusively trained on images within a fixed resolution range. These segmentations are inaccurate for very high-resolution images since using bicubic upsampling of low-resolution segmentation does not adequately capture high-resolution details along object boundaries. In this paper, we propose a novel approach to address the high-resolution segmentation problem without using any high-resolution training data. The key insight is our CascadePSP network which refines and corrects local boundaries whenever possible. Although our network is trained with low-resolution segmentation data, our method is applicable to any resolution even for very high-resolution images larger than 4K. We present quantitative and qualitative studies on different datasets to show that CascadePSP can reveal pixel-accurate segmentation boundaries using our novel refinement module without any finetuning. Thus, our method can be regarded as class-agnostic. Finally, we demonstrate the application of our model to scene parsing in multi-class segmentation.

In this paper, the filter-and-forward (FF) relay design for multiple-input-multiple-output (MIMO) orthogonal frequency-division multiplexing (OFDM) systems is investigated. Due to the considered MIMO structure, the FF relay design is investigated in the framework of joint design together with the linear MIMO transceiver. As the design criterion, first, the minimization of weighted sum mean square error (MSE) is considered. The joint design in this case is approached based on alternating optimization that iterates between the optimal design of the FF relay for given MIMO precoding and decoding matrices and the optimal design of MIMO precoding and decoding matrices for a given FF relay filter. Second, a more advanced problem of joint design for rate maximization is considered. The second problem is approached based on the obtained result regarding the first problem of weighted sum MSE minimization and the existing result regarding the relationship between weighted MSE minimization and rate maximization. Numerical results show the effectiveness of the proposed FF relay design method and significant performance improvement by the proposed FF relay over widely considered simple AF relays for MIMO-OFDM systems.