As an emerging interpretable technique, Generalized Additive Models (GAMs) adopt neural networks to individually learn non-linear functions for each feature, which are then combined through a linear model for final predictions. Although GAMs can explain deep neural networks (DNNs) at the feature level, they require large numbers of model parameters and are prone to overfitting, making them hard to train and scale. Additionally, in real-world datasets with many features, the interpretability of feature-based explanations diminishes for humans. To tackle these issues, recent research has shifted towards concept-based interpretable methods. These approaches try to integrate concept learning as an intermediate step before making predictions, explaining the predictions in terms of human-understandable concepts. However, these methods require domain experts to extensively label concepts with relevant names and their ground-truth values. In response, we propose CAT, a novel interpretable Concept-bAsed Taylor additive model to simplify this process. CAT does not require domain experts to annotate concepts and their ground-truth values. Instead, it only requires users to simply categorize input features into broad groups, which can be easily accomplished through a quick metadata review. Specifically, CAT first embeds each group of input features into one-dimensional high-level concept representation, and then feeds the concept representations into a new white-box Taylor Neural Network (TaylorNet). The TaylorNet aims to learn the non-linear relationship between the inputs and outputs using polynomials. Evaluation results across multiple benchmarks demonstrate that CAT can outperform or compete with the baselines while reducing the need of extensive model parameters. Importantly, it can effectively explain model predictions through high-level concepts. Source code is available at github.com/vduong143/CAT-KDD-2024.

As the volume of content and the connectivity of social media have grown, snackable content has increasingly become an enjoyable and engaging way to share content. Snackable content is a shortened form of original content focusing on a single theme or motif for entertainment and quick understanding of a video moment. For content owners with a large library of long-form content (movies, television series, documentaries, etc.), one challenge in accommodating snackable content in social media uses is the correct identification and cutting of interesting regions. Related problems have been studied for algorithmic discovery of content for movie trailers, short-duration meme content, and medium duration news stories, but none of these approaches included user preferences as explicit drivers for cuts. This paper analyzes both human and automatic methods for creating snackable clips across different categories of content with two comprehensive user studies. Contrary to initial expectations, findings amongst the surveyed population indicate a preference for slightly longer snackable clips (60-90 seconds) and those that began or ended with a human character.

Streaming media is gaining popularity, with numerous new services for video on demand and live broadcast. These services stream requested media content to user devices like smart TVs, personal computers and smartphones. Many of these devices are mobile devices, yet streaming media services do not adapt the delivered content to the context of the request, e.g., the location of the user, the time of the request or who are the people near the viewer. In this paper we illustrate how geofencing and geoblocking can be combined with adaptive streaming media to create a new technology of contextually-adapted streaming media. The new technology has the potential to create a new type of experience for streaming media by applying real-time modification of streamed content according to the viewer and the context, e.g., modify inappropriate content when the media is played in public places, change the length of a video played by a train passenger according to a train schedule, etc. The suggested technology can also be used for preventing password sharing in a non-intrusive way and for hyperlocal geoblocking, to facilitate copyright protection. We discuss the vision of modifying streaming media in real time based on the context, elaborate on some of the challenges in implementing this vision, and present novel applications of this new technology.

Combining content analysis of television programs with quantitative audience measurement can provide insights into customer reactions to advertisements and program content. This work introduces a system architecture that incorporates anonymous audience metrics from an operational IPTV environment with metadata from a content-based analysis of recorded programs. Evaluated on a collection of news programs, the system verifies that events derived from the audience metrics data stream correspond to media segmentation boundaries such as commercial breaks and topic changes. An automated system for executing multimodal media segmentation algorithms for commercial break and topic change detection is also discussed. Better understanding of audience reaction can help IPTV service providers plan infrastructure investments and help in managing multimedia content delivery networks.

Video-based lip activity analysis has been successfully used for assisting speech recognition for almost a decade. Surprisingly, this same capability has not been heavily used for near real-time visual speaker retrieval and verification, due to tracking complexity, inadequate or difficult feature determination, and the need for a large amount of pre-labeled data for model training. This paper explores the performance of several solutions using modern histogram of oriented gradients (HOG) features, several quantization techniques, and analyzes the benefits of temporal sampling and spatial partitioning to derive a representation called LipActs. Two datasets are used for evaluation: one with 81 participants derived from varying quality YouTube content and one with 3 participants derived from a forward facing mobile video camera with 10 varied lighting and capture angle environments. Over these datasets, LipActs with a moderate number of pooled temporal frames and multi-resolution spatial quantization, offer an improvement of 37-73% over raw features when optimizing for lowest equal error rate (EER).

Exploding amounts of multimedia data increasingly require automatic indexing and classification, e.g. training classifiers to produce high-level features, or semantic concepts, chosen to represent image content, like car, person, etc. When changing the applied domain (i.e. from news domain to consumer home videos), the classifiers trained in one domain often perform poorly in the other domain due to changes in feature distributions. Additionally, classifiers trained on the new domain alone may suffer from too few positive training samples. Appropriately adapting data/models from an old domain to help classify data in a new domain is an important issue. In this work, we develop a new cross-domain SVM (CDSVM) algorithm for adapting previously learned support vectors from one domain to help classification in another domain. Better precision is obtained with almost no additional computational cost. Also, we give a comprehensive summary and comparative study of the state-of-the-art SVM-based cross-domain learning methods. Evaluation over the latest large-scale TRECVID benchmark data set shows that our CDSVM method can improve mean average precision over 36 concepts by 7.5%. For further performance gain, we also propose an intuitive selection criterion to determine which cross-domain learning method to use for each concept.

The proposed shot boundary determination (SBD) algorithm contains a set of finite state machine (FSM) based detectors for pure cut, fast dissolve, fade in, fade out, dissolve, and wipe. Support vector machines (SVM) are applied to the cut and dissolve detectors to further boost performance. Our SBD system was highly effective when evaluated in TRECVID 2006 (TREC video retrieval evaluation) and its performance was ranked highest overall.