Action recognition and localization in videos pose significant challenges, as they require identifying temporal boundaries and classifying actions within long video sequences. This paper introduces an anchor-free, single-stage framework that predicts both the start and end times of actions while simultaneously classifying the actions, framing the task as a sequence labeling problem. The proposed model utilizes an encoder-decoder architecture with LSTM projections and 1D convolutions to capture rich temporal dependencies. This is followed by hierarchical feature pyramid generation, which is refined using Pyramidal Pooling Aggregation (PPA) and enhanced through a Progressive Feature Enhancement Unit (PFEU) with dilated convolutions to preserve the contextual relationships present in complex video frames. We also propose a novel Temporal Granularity Convolution (TGC) Layer, which refines temporal features. The TGC Layer captures various temporal details using a multi-branch structure consisting of Fine-Grained and Coarse-Grained Temporal Branches, designed to efficiently handle different temporal granularities. Finally, a decoder utilizes these multi-scale features to predict action instances at multiple levels of granularity. The model simultaneously integrates classification and regression heads to predict action labels and temporal boundaries, achieving improved localization and recognition. We have demonstrated the effectiveness of the proposed scheme, using mean average precision (mAP) across various Intersection over Union (IoU) thresholds on the Thumos14, ActivityNet-1.3, and MultiTHUMOS datasets over multiple state-of-the-art (SOTA) methods.

In this article, we proposed a technique for action localization and recognition from long untrimmed videos. It consists of C3D CNN model followed by the action mining using the localization model, where the KNN classifier is used. We segment the video into expressible sub-action known as action-bytes. The pseudo labels have been used to train the localization model, which makes the trimmed videos untrimmed for action-bytes. We present experimental results on the recent benchmark trimmed video dataset “Thumos14”.