Abstract In this technological era, human activity recognition (HAR) plays a significant role in several applications like surveillance, health services, Internet of Things, etc. Recent advancements in deep learning and video summarization have motivated us to integrate these techniques for HAR. This paper introduces a computationally efficient HAR technique based on a deep learning framework, which works well in realistic and multi-view environments. Deep convolutional neural networks (DCNNs) normally suffer from different constraints, including data size dependencies, computational complexity, overfitting, training challenges and vanishing gradients. Additionally, with the use of advanced mobile vision devices, the demand for computationally efficient HAR algorithms with the requirement of limited computational resources is high. To address these issues, we used integration of DCNN with video summarization using keyframes. The proposed technique offers a solution that enhances performance with efficient resource utilization. For this, first, we designed a lightweight and computationally efficient deep learning architecture based on the concept of identity skip connections (features reusability), which preserves the gradient loss attenuation and can handle the enormous complexity of activity classes. Subsequently, we employed an efficient keyframe extraction technique to minimize redundancy and succinctly encapsulate the entire video content in a lesser number of frames. To evaluate the efficacy of the proposed method, we performed the experimentation on several publicly available datasets. The performance of the proposed method is measured in terms of evaluation parameters Precision, Recall, F-Measure and Classification Accuracy. The experimental results demonstrated the superiority of the presented algorithm over other existing state-of-the-art methods.

Summary Human activity recognition (HAR) has gained researcher's interest due to its increasing demand in automated monitoring applications. Development of efficient HAR algorithm is still an open research area due to the challenges like inter and intra‐class variations, diversity in lighting conditions, view point changes, and complex object motions. Convolutional neural network (CNN) based methods have achieved significant improvement in HAR. However, CNN implementations have drawback that it require a lot of computational resources due to the use of large number of learnable parameters. To overcome this drawback, we propose a simple and computationally efficient deep CNN architecture using multi‐layer information fusion for HAR. In this study, we explore the impact of information fusion at intermediate layers of the network, as each convolutional layer of the network hierarchically extracts information at different level of abstraction of the objects from the video frames. In this work, first we designed a simple and computationally efficient deep CNN architecture and then we introduce a feature fusion strategy to integrate the complementary information of intermediate layers to the layer of the proposed CNN architecture. The proposed architecture is fine‐tuned and trained from scratch with raw RGB data. Softmax classifier is used at the last layer of network for activity classification. Benefits of the proposed architecture over standard deep learning architectures is it's high computational efficiency and reduced requirement of computational resources. To prove the effectiveness of the proposed method, we performed several extensive experiments on publically available datasets. The experimental results of the proposed method have demonstrated its superiority over other existing state‐of‐the‐art methods.

Segmentation of moving object in video with moving background is a challenging problem and it becomes more difficult with varying illumination. The authors propose a dense optical flow‐based background subtraction technique for object segmentation. The proposed technique is fast and reliable for segmentation of moving objects in realistic unconstrained videos. In the proposed work, they stabilise the camera motion by computing homography matrix, then they perform statistical background modelling using single Gaussian background modelling approach. Moving pixels are identified using dense optical flow in the background modelled scenario. The dense optical flow provides motion information of each pixel between consecutive frames, therefore for moving pixel identification they compute motion flow vector of each pixel between consecutive frames. To distinguish between foreground and background pixels, they labelled each pixel and thresholding the magnitude of motion flow vector identifies the moving pixels. The effectiveness of the proposed algorithm has been evaluated both qualitatively and quantitatively. The proposed algorithm has been evaluated on several realistic videos of different complex conditions. To assess the performance of the proposed work, the authors compared their algorithm with other state‐of‐art methods and found that the proposed method outperforms the other methods.

In the present article, we present an algorithm of content based video retrieval using Frame fusion and Histogram of Oriented Gradients (HOG). Representative frames of database videos are pre-processed using frame fusion to get a high resolution representative frames and HOG descriptor of this high resolution representative frames represents corresponding database video. On other side, query frame also undergo frame fusion and the HOG descriptor of high resolution query frame is used to represent query frame. To retrieve videos similar to query frame, matching is done using Euclidean distance between HOG features of query frame and database representative frames. The proposed method is tested on news category videos. The proposed method randomly picks frames from database videos, instead of selecting keyframes as query frames. Performance is assessed with the parameters precision, recall, accuracy and Jaccard index. The experimental results have shown that the performance of the proposed method is performing better than other state-of-art methods.

In recent years, research in the field of keyframe extraction become more attractive due to its use in advanced applications like video surveillance. In this paper, we introduce a novel algorithm of keyframe extraction which utilizes Binary Robust Invariant Scalable Keypoint features to obtain the dissimilarity level of consecutive frames and establishes shot transition boundary, from where we extract keyframes. The frame at which dissimilarity level is high is taken as a keyframe. The proposed algorithm is tested on ten different videos of animation category. Performance of the method is assessed using the evaluation metrics- Figure of merit, Detection percentage, Accuracy and missing factor. The experimental results and analysis shows improved performance of the proposed algorithm over the other state-ofthe-art methods.

With the growth of various image-capturing devices, image acquisition is no longer a difficult task. As this technology is flourishing, various types of complex images are being produced. In order to access a large number of images stored in database easily, the images must be properly organized. Field of image retrieval attempts to solve this problem. As the complex images are being produced, processing them using single-resolution techniques is not sufficient as these images may contain varying levels of details. This paper proposes a novel multiresolution descriptor, local binary curvelet co-occurrence pattern, to achieve the task of content-based image retrieval. Curvelet transform of grayscale image is computed followed by computation of local binary pattern of resulting curvelet coefficients. Finally, feature vector is constructed using grey-level co-occurrence matrix which is matched with the feature vector of database images. The proposed descriptor combines the properties of local pattern and multiresolution technique of curvelet transform, and efficiently covers curvilinear and geometrical structures present in the image. Performance of the proposed method is measured in terms of precision and recall and is tested on five benchmark datasets consisting of natural images. The proposed method has been compared with single and multiresolution techniques as well as with some of the other state-of-the-art image retrieval methods. The experimental results clearly demonstrate that the proposed method produces high retrieval accuracy and outperforms other techniques in terms of precision and recall.

Various multimodal medical images like computed tomography (CT), magnetic resonance imaging (MRI), positron emission tomography, single photon emission CT and structural MRI have different characteristics and carry different types of complementary anatomical and functional information. Therefore, fusion of multimodal images is required, in order to achieve good spatial resolution images carrying both anatomical and functional information. In this work, the authors have proposed a fusion technique based on curvelet transform. Curvelet transform is a multiscale, multidirectional transform having anisotropic property and is very efficient in capturing edge points in images. Edges in an image are the important information carrying points used to show better visual structure of the image. They use local energy‐based fusion rule which is more effective than single pixel‐based fusion rules. Comparison of the proposed method with other existing spatial and wavelet transform based methods, in terms of visual and quantitative measures show the effectiveness of the proposed method. For quantitative analysis of the method, they used five fusion metrics as entropy, standard deviation, edge‐strength, sharpness and average gradient.

Shadow detection and removal is an important problem in computer vision. The real challenge in moving shadow detection and removal is to classify moving shadow points which are many times misclassified as moving object points in a video sequences. Various shadow detection and removal algorithms have been proposed for images but only a few works have been done for moving objects. In this study, a novel method for shadow detection and removal is proposed using discrete wavelet transform (DWT). The authors have used DWT because of its multi‐resolution property that decomposes an image into four different bands without loss of the spatial information. For detection and removal of shadow, they have proposed a new threshold in the form of relative standard deviation. The value of threshold is automatically determined and does not require any supervised learning or manual calibration. The proposed method is flexible and depends on only one parameter, namely, wavelet coefficients. Results of shadow detection and removal from moving object after applying the proposed method are compared with the results of other state‐of‐the‐art methods in terms of visual performance and a number of quantitative performance parameters. The proposed method is found to be better and more robust than other methods.

Research in motion analysis is a challenging field and it has a variety of video surveillance applications. For any video surveillance application, background detection and removal plays an important role in segmentation of the moving objects. This study proposes a new method for segmentation of the moving object, which is based on single change detection applied on Daubechies complex wavelet coefficients of two consecutive frames. The authors have chosen Daubechies complex wavelet transform as it is shift invariant and has a better directional selectivity as compared with real‐valued wavelet transforms. Single change detection is a method to obtain video object plane by inter‐frame difference of two consecutive frames, and it provides automatic detection of appearances of new objects. The proposed method does not require any other parameter except wavelet coefficients. Segmentation results of the moving objects after applying the proposed method are compared with those obtained after applying other spatial and wavelet domain segmentation methods in terms of visual performance and a number of quantitative measures viz misclassification penalty, relative position‐based measure, structural content, normalised absolute error and average difference and the proposed method is found better than the other methods.

In this paper, we have proposed a new method for segmentation of moving objects, which is based on single change detection applied on curvelet coefficients of two consecutive frames. The wavelet transform is widely used in moving object segmentation but it can not describe curve discontinuities. Therefore we have used curvelet transform for segmentation of moving objects. The proposed method is simple and does not require any other parameter except curvelet coefficients. Results after applying the proposed method for segmentation of moving object are compared with other state-of-the-art methods in terms of visual as well as quantitative performance measures viz. Misclassification penalty, Relative position based measure and Structural content. The proposed method is found to be better than other methods.

Multiclass object classification is a difficult problem in computer vision application, because of highly variable nature of different objects. The primary goal of this paper is to classify object into one of the chosen classes. The proposed method uses Dual tree complex wavelet transform coefficients as a feature of object. Dual tree complex wavelet transform is having advantage of its better edge representation and approximate shift-invariant property as compared to real valued wavelet transform. We have used multiclass support vector machine classifier for classification of objects. The proposed method has been tested on dataset prepared by authors of this paper. We have tested the proposed method on multiple levels of Dual tree complex wavelet transform. Quantitative evaluation results demonstrate that the proposed method gives better performance for multiclass object classification in comparison to other state-of-the-art methods.