Large enterprise IT (Information Technology) infrastructure components generate large volumes of alerts and incident tickets. These are manually screened, but it is otherwise difficult to extract information automatically from them to gain insights in order to improve operational efficiency. We propose a framework to cluster alerts and incident tickets based on the text in them, using unsupervised machine learning. This would be a step towards eliminating manual classification of the alerts and incidents, which is very labor intense and costly. Our framework can handle the semi-structured text in alerts generated by IT infrastructure components such as storage devices, network devices, servers etc., as well as the unstructured text in incident tickets created manually by operations support personnel. After text pre-processing and application of appropriate distance metrics, we apply different graph-theoretic approaches to cluster the alerts and incident tickets, based on their semi-structured and unstructured text respectively. For automated interpretation and read-ability on semi-structured text clusters, we propose a method to visualize clusters that preserves the structure and human-readability of the text data as compared to traditional word clouds where the text structure is not preserved; for unstructured text clusters, we find a simple way to define prototypes of clusters for easy interpretation. This framework for clustering and visualization will enable enterprises to prioritize the issues in their IT infrastructure and improve the reliability and availability of their services.

Similarity search, namely, finding approximate nearest neighborhoods, is the core of many large scale machine learning or vision applications. Recently, many research results demonstrate that hashing with compact codes can achieve promising performance for large scale similarity search. However, most of the previous hashing methods with compact codes only model and optimize the search accuracy. Search time, which is an important factor for hashing in practice, is usually not addressed explicitly. In this paper, we develop a new scalable hashing algorithm with joint optimization of search accuracy and search time simultaneously. Our method generates compact hash codes for data of general formats with any similarity function. We evaluate our method using diverse data sets up to 1 million samples (e.g., web images). Our comprehensive results show the proposed method significantly outperforms several state-of-the-art hashing approaches.

The increasingly large amount of digital multimedia content has created a need for technologies to search and identify multimedia files. Multimedia fingerprinting has been widely researched and successfully commercialized as a technology to trace and recognize both audio and video content. Most published research on multimedia fingerprinting focuses on the design of the fingerprinting extraction algorithm. The architecture of the fingerprinting database and the database query search algorithm has only been investigated in very few publications. In particular, no comprehensive mathematical models describing the accuracy of database query algorithms have been developed. Therefore, most performance evaluations of fingerprint system rely on experimental data. In this paper, we develop an analytic model to describe the operation of hash-table based multimedia fingerprinting databases. It allows to predict the accuracy of a fingerprinting query algorithm as a function of the statistical distribution of the fingerprints and the specific design of the fingerprint database.

We propose three methods to derive longer fingerprints from features using projection based hashing methods. For this class of hashing methods, a feature matrix is projected onto a set of projection matrices. Then, the projected values are compared to a threshold to derive the hash bits. The methods proposed in this paper reduce the increase in average number of collisions per hour of content in the database by creating longer fingerprint codewords. The first method derives 56 fingerprint bits by projecting the feature matrix onto 56 projection matrices by deriving one bit from each projected value. The second method uses 36 projection matrices but derives 56 fingerprint bits by deriving varying number of bits from each projected value. The third method also 36 projection matrices but derives 56 bits by creating more features that capture the relationships between the projected values. We present experimental results comparing these three methods to a baseline method in terms of collision performance and robustness.

In this paper, we propose two video fingerprinting methods that are robust to both geometric and non-geometric modifications on content. Both of the proposed methods are based on computation of moment invariants as features from concentric circular regions. The two methods differ in the way they capture appearance and motion information from video. In one method, we capture motion information by computing a difference image between the current video frame and a temporal average video frame computed from a past window of video frames. This method captures appearance by computing moment invariants from concentric circular regions of a video frame. In the second method, we capture appearance and motion by projecting features onto two sets of basis functions and explicitly capture how the moment invariants change over the regions and over time. We present experimental results on both of these video fingerprinting comparing their performance in terms of robustness against attacks and sensitivity to content.

In this paper, we study the collision property of one of the robust hash functions proposed. This method was originally proposed for robust hash generation from blocks of image data and is based on projection of image block data on pseudo-random matrices. We show that collision performance of this robust hash function is not optimal when used to extract hash bits from a moment invariants feature matrix for video fingerprinting. We identify that the collision performance of this hash extraction method could be improved if the pseudo-random matrices are selected carefully. We propose two methods that use an offline training set to improve the collision property. Both of the methods attempt to select the matrices that minimize cross-correlation among the projected features. The first method uses an iterative procedure to select the matrices that satisfy a cross-correlation threshold. The second method used Singular Value Decomposition (SVD) of the feature covariance matrix and hence the crosscorrelation of the projected values is zero. We show the improved collision performance of both these methods on the same dataset. Also, we interpret the projection matrices obtained through the SVD procedure and show that they capture appearance and motion information from the moment invariants feature matrix.

We present a novel video fingerprinting method based on subspace embedding. The proposed method is particularly robust against frame-rate conversion attacks and geometric attacks among other attacks including compression and spatial scaling. Using a sliding window, we extract fingerprints from a group of subsequent video frames. For the generation of the fingerprints, we first calculate the basis vectors of a coarse representation of this group of frames using a singular value decomposition (SVD). Then, we project the coarse representation of the video frames onto a subset of the basis vectors. Thus, we obtain a subspace representation of the input video frames. Finally, we extract the fingerprint bits by projecting a temporal average of these representations onto pseudorandom basis vectors. Since the subspace is estimated from the input video data itself, any global attack on video such as rotation would result in a corresponding change in estimated basis vectors thereby preserving the subspace representation. We present experimental results on 250 hrs of video to show the robustness and sensitivity of the proposed signature extraction method.

We propose a framework based on signatures extracted from audio and video streams for automatically measuring and maintaining synchronization between the two streams. The audio signature is based on projections of a coarse representation of the spectrogram onto random vectors. The video signature is based on projections of a coarse representation of the difference image between two consecutive frames onto random vectors. The time alignment present at the signature generator between the two streams is recorded by combining audio and video signatures into a combined synchronization signature. At the detector after video and audio streams go through different processing operations, we extract the signatures again. The signatures extracted before and after processing from the audio and the video are compared independently using a Hamming distance based correlator to estimate the relative misalignment introduced due to processing in each of the streams. Then, the estimated relative misalignment between the audio and video streams is used to preserve the same alignment between the streams that was present before processing. Our experimental results show that we can achieve > 93.0% accuracy in synchronization.

Content-based signatures are designed to be a robust bit-stream representation of the content so as to enable content identification even though the original content may go through various signal processing operations. In this paper, we propose a novel content-based audio signature extraction method that captures temporal evolution of the audio spectrum. The proposed method, first, divides the input audio into overlapping chunks and computes a spectrogram for each chunk. Then, it projects each of the spectrograms onto random basis vectors to create a signature that is a low-dimensional bit-stream representation of the corresponding spectrogram. Our experimental results show the robustness and sensitivity of the proposed content-based audio signature extraction method for various signal processing operations on audio content.

We propose a novel video signature extraction method based on projections of difference images between consecutive video frames. The difference images are projected onto random basis vectors to create a low dimensional bitstream representation of the active content (moving regions) between two video frames. A sequence of these signatures serves to identify the underlying video content in a robust manner. Our experimental results show that the proposed video signature is robust to most common signal processing operations on video content such as compression, resolution scaling, brightness scaling.

The problem of generative process tracking involves detecting and adapting to changes in the underlying generative process that creates a time series of observations. It has been widely used for visual background modelling to adaptively track the generative process that generates the pixel intensities. In this paper, we extend this idea to audio background modelling and show its applications in surveillance domain. We adaptively learn the parameters of the generative audio background process and detect foreground events. We have tested the effectiveness of the proposed algorithms using synthetic time series data and show its performance on elevator audio surveillance

In the recent years, consumer devices that can record broadcast video have become prevalent. Such devices rely on the program guide information about a program start time and end time for recording. However, sports broadcasts can run over the specified time sometimes. In this paper, we propose a framework based on an audio classification framework to correctly detect the end of a sport broadcast so as to enable complete recording of the game. Our experimental results show that the proposed algorithm can detect sports program boundaries with a high accuracy

We propose to use a visual object (e.g., the baseball catcher) detection algorithm to find local, semantic objects in video frames in addition to an audio classification algorithm to find semantic audio objects in the audio track for sports highlights extraction. The highlight candidates are then further grouped into finer-resolution highlight segments, using color or motion information. During the grouping phase, many of the false alarms can be correctly identified and eliminated. Our experimental results with baseball, soccer and golf video are promising.

We examine the significance of video mining as pattern discovery in multimedia content. We examine the underlying issue of pattern discovery versus pattern recognition, since most past work has not drawn such a sharp distinction. We argue that while the term "pattern discovery" implies a purely unsupervised approach, in practice a mixture of unsupervised and supervised techniques will have to be used. We compare conventional data mining with video mining and observe that a key difference is in the multilayered semantics of multimedia content. We then identify significant challenges posed by video mining.

Pattern discovery from video has promising applications in summarizing different genre types, including surveillance and sports. After pattern discovery, a summary of the video can be constructed from a combination of usual and unusual patterns, depending on the application domain. Previously, we used an unsupervised label mining approach to extract highlight moments from soccer videos (Radhakrishan, R. et al., IEEE Pacific-Rim Conf. on Multimedia, 2003). We now formulate the problem of pattern discovery from semantic audio labels as a time series clustering problem and propose a new unsupervised mining framework based on segmentation theory using eigenvectors of the affinity matrix. We test the validity of the technique using synthetically generated label sequences as well as label sequences from broadcast sports video. Our sports highlights extraction accuracy is comparable to that achieved in our previous work.

In fitting the training data with Gaussian mixture models (GMMs) of appropriate structures using the MDL (minimum description length) criterion, we are able to improve audio classification accuracy with a large margin. With the MDL-GMMs, we are also able to greatly improve the accuracy in extracting sports highlights. Since we have focused on audio domain processing, it enables us to extract highlights very quickly. We have demonstrated the importance of a better understanding of model structures in such a pattern recognition task.

We present a comparison of 6 methods for classification of sports audio. For feature extraction, we have two choices: MPEG-7 audio features and Mel-scale frequency cepstrum coefficients (MFCC). For classification, we also have two choices: maximum likelihood hidden Markov models (ML-HMM) and entropic prior HMMs (EP-HMM). EP-HMMs, in turn, have two variations: with and without trimming of the model parameters. We thus have 6 possible methods, each of which corresponds to a combination. Our results show that all the combinations achieve classification accuracy of around 90% with the best and the second best being, respectively, MPEG-7 features with EP-HMM and MFCC with ML-HMM.

We have developed a unified framework to extract highlights from three sports - baseball, golf and soccer - by detecting some of the common audio events that are directly indicative of highlights. We used MPEG-7 audio features and entropic prior hidden Markov models (HMM) for feature extraction and classification, respectively, to recognize these common audio events. Together with pre- and post-processing techniques using general sports knowledge, we have been able to generate promising results dealing with an audio track that is dominated by audio mixtures and noisy background.

In our past work we have used temporal patterns of motion activity to extract sports highlights. We have also used audio classification based approaches to develop a common audio-based platform for feature extraction that works across three different sports. In this paper, we combine the two aforementioned complementary approaches so as to get higher accuracy. We propose a framework for mining the semantic audio-visual labels in order to detect "interesting" events. Our results show that the proposed techniques work well across our three sports of interest, soccer, golf and baseball.

We present a comparison of 6 methods for classification of sports audio. For the feature extraction we have two choices: MPEG-7 audio features and Mel-scale frequency cepstrum coefficients (MFCC). For the classification we also have two choices: maximum likelihood hidden Markov models (ML-HMM) and entropic prior HMM (EP-HMM). EP-HMM, in turn, has two variations: with and without trimming of the model parameters. We thus have 6 possible methods, each of which corresponds to a combination. Our results show that all the combinations achieve classification accuracy of around 90% with the best and the second best being MPEG-7 features with EP-HMM and MFCC with ML-HMM.

We developed a unified framework to extract highlights from three sports: baseball, golf and soccer by detecting some of the common audio events that are directly indicative of highlights. We used MPEG-7 audio features and entropic prior hidden Markov models (HMM) as the audio features and classifier respectively to recognize these common audio events. Together with pre- and post-processing techniques using general sports knowledge, we have been able to generate promising results dealing with the audio track that is dominated by audio mixtures and noisy background.

We describe a key-frame extraction technique based on the intuition that the higher the motion the more the number of key-frames required for summarization. We verify experimentally that the intensity of motion activity directly indicates the summarizability of the video segment, by using the MPEG-7 motion activity descriptor (see Jeannin, S. and Divakaran, A., IEEE Trans. Circuits and Systems for Video Tech., vol.11, no.6, p.720-4, 2001) and the fidelity measure described by H.S. Chang et al. (see IEEE Trans. Circuits and Systems for Video Tech., vol.9, no.8, p.1269-79, 1999). We obtain the key-frames by dividing the shot in parts of equal cumulative motion activity, and then selecting the frame located at the half-way point of each sub-segment. Furthermore, we establish an empirical relationship between the motion activity of a segment and the required number of key-frames. We thus provide a unique and rapid way to find the required number of key-frames and compute them. Our scheme is much faster than conventional color-based key-frame extraction schemes since it relies on simple computation and compressed domain extraction. It is close to the theoretical optimum in accuracy.

We investigate the image authentication system SARI, proposed by Lin and Chang (see ibid., vol.11, p.153-68, Feb. 2001), that distinguishes JPEG compression from malicious manipulations. In particular, we took at the image digest component of this system. We show that if multiple images have been authenticated with the same secret key and the digests of these images are known to an attacker, Oscar, then he can cause arbitrary images to be authenticated with this same but unknown key. We show that the number of such images needed by Oscar to launch a successful attack is quite small, making the attack very practical. We then suggest possible solutions to enhance the security of this authentication system.

We investigate the image authentication system, SARI, proposed by C.Y. Lin and S.F. Chang (see SPIE Storage and Retrieval of Image/Video Databases, 1998), that distinguishes JPEG compression from malicious manipulations. In particular, we look at the image digest component of this system. We show that if multiple images have been authenticated with the same secret key and the digests of these images are known to an attacker, Oscar, then he can cause arbitrary images to be authenticated with this same but unknown key. We show that the number of such images needed by Oscar to launch a successful attack is quite small, making the attack very practical. We then suggest possible solutions to enhance the security of this authentication system.