We introduce VisualCent, a unified human pose and instance segmentation framework to address generalizability and scalability limitations to multi-person visual human analysis. VisualCent leverages centroid-based bottomup keypoint detection paradigm and uses Keypoint Heatmap incorporating Disk Representation and KeyCentroid to identify the optimal keypoint coordinates. For the unified segmentation task, an explicit keypoint is defined as a dynamic centroid called MaskCentroid to swiftly cluster pixels to specific human instance during rapid changes in human body movement or significantly occluded environment. Experimental results on COCO and OCHuman datasets demonstrate VisualCent’s accuracy and real-time performance advantages, outperforming existing methods in mAP scores and execution frame rate per second. The implementation is available on the project page†.†https://sites.google.com/view/niazahmad/projects/visualcent

The dynamic movement of the human body presents a fundamental challenge for human pose estimation and body segmentation. State-of-the-art approaches primarily rely on combining keypoint heatmaps with segmentation masks, but often struggle in scenarios involving overlapping joints during pose estimation or rapidly changing poses for instance-level segmentation. To address these limitations, we leverage Keypoints as Dynamic Centroid (KDC), a new centroid-based representation for unified human pose estimation and instance-level segmentation. KDC adopts a bottom-up paradigm to generate keypoint heatmaps for easily distinguishable and complex keypoints, and improves keypoint detection and confidence scores by introducing KeyCentroids using a keypoint disk. It leverages high-confidence keypoints as dynamic centroids in the embedding space to generate MaskCentroids, allowing for the swift clustering of pixels to specific human instances during rapid changes in human body movements in a live environment. Our experimental evaluations focus on crowded and occluded cases using the CrowdPose, OCHuman, and COCO benchmarks, demonstrating KDC’s effectiveness and generalizability in challenging scenarios in terms of both accuracy and runtime performance. Our implementation is available at https://sites.google.com/view/niazahmad/projects/kdc.

Despite the advances in multi-person pose estimation, state-of-the-art techniques only deliver the human pose structure.Yet, they do not leverage the keypoints of human pose to deliver whole-body shape information for human instance segmentation. This paper presents PosePlusSeg, a joint model designed for both human pose estimation and instance segmentation. For pose estimation, PosePlusSeg first takes a bottom-up approach to detect the soft and hard keypoints of individuals by producing a strong keypoint heat map, then improves the keypoint detection confidence score by producing a body heat map. For instance segmentation, PosePlusSeg generates a mask offset where keypoint is defined as a centroid for the pixels in the embedding space, enabling instance-level segmentation for the human class. Finally, we propose a new pose and instance segmentation algorithm that enables PosePlusSeg to determine the joint structure of the human pose and instance segmentation. Experiments using the COCO challenging dataset demonstrate that PosePlusSeg copes better with challenging scenarios, like occlusions, en-tangled limbs, and overlapped people. PosePlusSeg outperforms state-of-the-art detection-based approaches achieving a 0.728 mAP for human pose estimation and a 0.445 mAP for instance segmentation. Code has been made available at: https://github.com/RaiseLab/PosePlusSeg.

Multi-person pose estimation and instance segmentation suffer large performance loss when images are with an increasing number of people and their uncontrolled complex appearance. Yet, existing models cannot efficiently leverage unbalanced training images, i.e., few of them are with multi-person, and most are with single-person, making them ineffective for challenging multi-person scenarios. To tackle multi-person cases with a limited portion of them, we propose MultiPoseSeg, a data preparation and feedback knowledge transfer system designed for multi-person pose estimation and instance segmentation. First, MultiPoseSeg categorically performs random data reduction to reduce the single-person bias from the train dataset. Second, MultiPoseSeg employs the knowledge transfer from ancestor models to converge the model learning with a limited amount of data and time. This way, our model learns and train on human pose and instance segmentation to advance the training and testing accuracy. Finally, MultiPoseSeg proposes keypoint maps to identify the keypoint coordinates for soft and hard keypoints and segmentation maps to assign centroid to each human instance, which helps to cluster the pixels to a particular instance. We have evaluated MultiPoseSeg using COCO and OCHuman challenging datasets and demonstrated MultiPoseSeg outperforms state-of-the-art bottom-up models in terms of both accuracy and runtime performance, achieving 0.728 mAP for pose and 0.445 mAP for segmentation on COCO dataset. All the unbiased data and code has been made available at: https://github.com/RaiseLab/MultiPoseSeg

The adaptation of deep convolutional neural network has made revolutionary advances in human body posture estimation. Various applications utilizing deep neural network for pose estimation have drawn considerable attention in recent years. However, prediction and localization of keypoints in single-person and multi-person images is still a challenging problem. Towards this, we propose a bottom-up approach to pose estimation and motion recognition. We present StrongPose system that deals with object-part associations using part-based modeling. The convolution network in our model detects strong keypoint heat maps and predicts their comparative displacements, allowing keypoints to be grouped into human instances. Further, it utilizes the keypoints to generate body heat maps that can determine the position of the human body in the image. The StrongPose system is based on fully convolutional engineering and makes proficient inferences while maintaining runtime regardless of the number of individuals in the image. We train and test the StrongPose on the COCO dataset. Evaluation results show that our framework achieves average precision of 0.708 using ResNet-101 and 0.725 using ResNet-152. Our results considerably outperform prior bottom-up frameworks.