To improve vision-based methods for understanding how people interact with their physical environment, we introduce a multi-view video and body-contact sensing dataset designed to capture dynamic human activities that involve interactions with the physical environment. The dataset includes activities such as parkour, physical training, and gym exercises, characterized by frequent body-environment contact. The proposed dataset includes 780 K images across 120 K pose sequences from 7 subjects. Each subject is captured by 6 synchronized third-person cameras, a single egocentric camera, and multiple contact sensors worn on the body. Using our proposed dataset, we benchmark state-of-the-art vision-based body contact models and show significant limitations in existing methods. Furthermore, we benchmark existing human pose estimation methods on our dataset and show that they fail under significant occlusion caused by close interactions with the environment, which indicates that our dataset can also be used to further develop pose estimation models to be more robust during interaction with the environment. To facilitate better human pose estimation from video, we introduce and evaluate a video-based human contact detection model that outperforms existing imagebased methods, underscoring the potential improvements from integrating contact information into pose estimation models. See the project page at: https://yohanshin.github.io/bodycontact4d.github.io/

The pandemic has contributed to the increased digital content development for remote experiences. Notably, museums have begun creating virtual exhibitions using 360-videos, providing a sense of presence and high level of immersion. However, 360-video content often uses a linear timeline interface that requires viewers to follow the path decided by the video creators. This format limits viewers’ ability to actively engage with and explore the virtual space independently. Therefore, we designed a map-based video interface, Viewer2Explorer, that enables the user to perceive and explore virtual spaces autonomously. We then conducted a study to compare the overall experience between the existing linear timeline and map interfaces. Viewer2Explorer enhanced users’ spatial controllability and enabled active exploration in virtual museum exhibition spaces. Additionally, based on our map interface, we discuss a new type of immersion and assisted autonomy that can be experienced through a 360-video interface and provide design insights for future content.

MOTIVATION: Protein-ligand binding affinity prediction is a central task in drug design and development. Cross-modal attention mechanism has recently become a core component of many deep learning models due to its potential to improve model explainability. Non-covalent interactions (NCIs), one of the most critical domain knowledge in binding affinity prediction task, should be incorporated into protein-ligand attention mechanism for more explainable deep drug-target interaction models. We propose ArkDTA, a novel deep neural architecture for explainable binding affinity prediction guided by NCIs. RESULTS: Experimental results show that ArkDTA achieves predictive performance comparable to current state-of-the-art models while significantly improving model explainability. Qualitative investigation into our novel attention mechanism reveals that ArkDTA can identify potential regions for NCIs between candidate drug compounds and target proteins, as well as guiding internal operations of the model in a more interpretable and domain-aware manner. AVAILABILITY: ArkDTA is available at https://github.com/dmis-lab/ArkDTA. CONTACT: [email protected].

Adaptive streaming service nowadays, became an essential key technology for delivering videos over internet protocol network. Due to limitations and fluctuations in internet bandwidth, scaler has become essential for streaming service. Recently, learning-based scalers have greatly improved performance compared to conventional methods in visual quality. However, performance is not guaranteed for compatibility of conventional scaler and real-time processing is difficult due to high complexity. In this paper, we present a low complexity scaler based on convolutional neural networks called Video Scale Network (VSN). The proposed method has a simple structure with real-time processing and a loss function compatible of conventional scaler. Furthermore, we propose a learning method to improve performance using only a single network. Experimental results on the AOM test sequences reveal improved performance by using the proposed method compared to the conventional method and real-time processing in the decoder of AV1 codec is also able to be done.

The goal of downscaling an image is to reduce its resolution to a lower resolution, while maintaining the visual characteristics of the original image. Recent learning-based algorithms have shown great improvement over conventional methods in preserving high-frequency information. However, they continue to suffer from various artifacts due to the lack of true high-low resolution image pairs in their training datasets. In this paper, an unsupervised image downscaler that preserves the high frequency content of the original image based on an autoencoder is presented. Specifically, the image downscaler is obtained by extracting the decoder of the developed autoencoder. Furthermore, we propose a preprocessing step that enables image downscaler to any arbitrary scales. Experimental results on five benchmark datasets reveal the qualitative and quantitative superiority of the proposed method at various scales compared to other methods.

Many museums have begun to use mobile devices to provide interactive content to visitors. However, such devices can also easily distract visitors, drawing their attention away from the exhibits. In this pictorial, we show the steps in designing Scienscope-a hand-held mediation device that can be used as a stethoscope to explore additional information hidden within exhibits at close range. As Scienscope hovers on authentic objects, it provides digital content and audio-haptic feedback, reinforcing the visitors' exploratory behaviors. After interviewing museum experts and bodystorming the museum space, we followed an iterative design process using low-fi prototyping to explore the users' experience. Three application scenarios were devised to show how Scienscope can be used for different types of exhibits. We discussed how visitors respond to and explore hidden information about exhibits at close range, which can be helpful for museum curators and designers.