Given a support image with keypoint annotations, a category-agnostic pose estimation (CAPE) method aims to predict keypoints in a query image from the same category. The relationship between keypoint features of the support image and the visual features of the query image is discovered to achieve CAPE. Although many CAPE methods have been proposed, two challenges remain: 1) Support information is limited, especially when only a few shots of support images are given; 2) the pose in the support image may be significantly different from that in the query image. For the first issue, we propose to consider keypoint features from the categories in the same super-category as the support image to enhance the support representations. For the second issue, we propose a query-adaptive adjacency matrix generated based on the similarity between features of support keypoints and the query image. Experimental results verify that both ideas bring performance gain and make the proposed method achieve state-of-the-art results on the MP-100 benchmark.

For table tennis videos, we develop a system to analyze and generate statistics based on ball trajectories. By a real-time ball detector, the ball trajectory is constructed based on the tracking by detection scheme. Landing points on the table are estimated. Based on moving direction and the sequence of landing points, three-stage analysis can be achieved. We also analyze how a point starts (serving type classification) and how a point ends (point loss classification).

Existing methods for manga character re-identification primarily rely on facial information, overlooking the unique characteristics of characters’ bodies and failing to address common challenges like occlusion by speech balloons and incomplete body parts. To tackle these issues, we propose a method called Occlusion-Aware Manga Character Re-identification (OAM-ReID) with self-paced contrastive learning, which leverages annotated body data from the Manga109 dataset for training. By synthesizing data with occluded speech balloons and incomplete bodies, we empower the framework to be aware of occlusion, so that more effective feature representations are learnt. Experimental results show that this approach outperforms the state-of-the-art person ReID method.

We propose a cross-modal attention module to combine information from different cues and different modalities, to achieve action recognition in still images. Feature maps are extracted from the entire image, the detected human bounding box, and the detected human skeleton, respectively. Inspired by the transformer structure, we design the processing between the query vector from one cue/modality, and the key vector from another cue/modality. Feature maps from different cues/modalities are cross-referred so that better representations can be obtained to yield better performance. We show that the proposed framework outperforms the state-of-the-art systems without the requirement of an extra training dataset. We also conduct ablation studies to investigate how different settings impact the final results.

Tactic learning in virtual reality (VR) has been proven to be effective for basketball training. Endowed with the ability of generating virtual defenders in real time according to the movement of virtual offenders controlled by the user, a VR basketball training system can bring more immersive and realistic experiences for the trainee. In this article, an autoregressive generative model for instantly producing basketball defensive trajectory is introduced. We further focus on the issue of preserving the diversity of the generated trajectories. A differentiable sampling mechanism is adopted to learn the continuous Gaussian distribution of player position. Moreover, several heuristic loss functions based on the domain knowledge of basketball are designed to make the generated trajectories assemble real situations in basketball games. We compare the proposed method with the state-of-the-art works in terms of both objective and subjective manners. The objective manner compares the average position, velocity, and acceleration of the generated defensive trajectories with the real ones to evaluate the fidelity of the results. In addition, more high-level aspects such as the empty space for offender and the defensive pressure of the generated trajectory are also considered in the objective evaluation. As for the subjective manner, visual comparison questionnaires on the proposed and other methods are thoroughly conducted. The experimental results show that the proposed method can achieve better performance than previous basketball defensive trajectory generation works in terms of different evaluation metrics.

To provide rich viewing experience and assist pitcher training, we propose an automatic baseball pitch overlay system in this paper. Given multiple pitching video sequences, this system detects and tracks the ball to construct ball trajectories. Because of occlusion, motion blur, and background noise, the ball usually cannot be detected successfully. We propose a series of processes like initial compensation and polynomial fitting to construct complete trajectories. To make the overlay results more appealing, different sequences are weighted differently, and different trajectories are intentionally drawn in different colors. We believe this would be the first fully-automatic pitch overlay system that only takes pitching videos as inputs. Source code is at \\https://github.com/chonyy/ML-auto-baseball-pitching-overlay.

The International Joint Workshop on Multimedia Artworks Analysis and Attractiveness Computing in Multimedia (MMArt-ACM) solicits contributions on methodology advancement and novel applications of multimedia artworks and attractiveness computing that emerge in the era of big data and deep learning. Despite the strike of the Covid-19 pandemic, this workshop attracts submissions of diverse topics in these two fields, and the workshop program finally consists of five presented papers. The topics cover image retrieval, image transformation and generation, recommendation system, and image/video summarization. The actual MMArt-ACM'20 Proceedings are available in the ACM DL at: https://dl.acm.org/citation.cfm?id=3379173

Learning basketball tactic via virtual reality environment requires real-time feedback to improve the realism and interactivity. For example, the virtual defender should move immediately according to the player's movement. In this paper, we proposed an autoregressive generative model for basketball defensive trajectory generation. To learn the continuous Gaussian distribution of player position, we adopt a differentiable sampling process to sample the candidate location with a standard deviation loss, which can preserve the diversity of the trajectories. Furthermore, we design several additional loss functions based on the domain knowledge of basketball to make the generated trajectories match the real situation in basketball games. The experimental results show that the proposed method can achieve better performance than previous works in terms of different evaluation metrics.

Among various elements of manga, character's face plays one of the most important role in access and retrieval. We propose a DNN-based method to do manga face detection, which is a challenging but relatively unexplored topic. Given a manga page, we first find candidate regions based on the selective search scheme. A deep neural network is then proposed to detect manga faces of various appearance. We evaluate the proposed method based on a large-scale benchmark, and show performance comparison and convincing evaluation results that have rarely done before.

Most of the broadcasted sports events nowadays present game statistics to the viewers which can be used to design the gameplay strategy, improve player's performance, or improve accessing the point of interest of a sport game. However, few studies have been proposed for broadcasted badminton videos. In this paper, we integrate several visual analysis techniques to detect the court, detect players, classify strokes, and classify the player's strategy. Based on visual analysis, we can get some insights about the common strategy of a certain player. We evaluate performance of stroke classification, strategy classification, and show game statistics based on classification results.

The emergence of cloud computing provides an unlimited computation/storage for users, and yields new opportunities for multimedia analysis and retrieval research. However, privacy of users, e.g., search intention, may be leaked to the server and maliciously utilized by companies or individuals with animus. This paper presents a privacy-preserving multimedia analysis framework based on a widely-adopted structure, i.e., bipartite graph, so that multimedia analysis and retrieval in the encrypted domain is enabled. This work aims to keep the server unaware of what the user wants to retrieve, and at the same time take advantage of the server's computation power. Homomorphic encryption schemes and communication protocols in the encrypted domain are integrated to facilitate bipartite graph construction and implement the Hungarian algorithm to find the best matching. Two applications, video tag suggestion and video copy detection, are developed on top of the privacy-preserving framework, and the evaluation results demonstrate that performance obtained in the encrypted domain is comparable with that obtained in the plain text domain.

We design a method to incorporate color information into the framework of CENsus Transform histogram (CENTRIST), a state-of-the-art visual descriptor for scene categorization. The newly proposed color CENTRIST descriptor describes global shape information by not only gradient derived from intensity values but also color variations between pixels in local image patches. Through extensive evaluations on various datasets, we demonstrate that the color CENTRIST descriptor is not only easily to be implemented, but also reliably achieves performance over that of CENTRIST.

Many objects have repetitive elements, and finding repetitive patterns facilitates object recognition and numerous applications. We devise a representation to describe configurations of repetitive elements. By modeling spatial configurations, visual patterns are more discriminative than local features, and are able to tackle with object scaling, rotation, and deformation. We transfer the pattern discovery problem into finding frequent subgraphs from a graph, and exploit a graph mining algorithm to solve this problem. Visual patterns are then exploited in architecture image classification and product image retrieval, based on the idea that visual pattern can describe elements conveying architecture styles and emblematic motifs of brands. Experimental results show that our pattern discovery approach has promising performance and is superior to the conventional bag-of-words approach.