Automatic recognition of human actions in video has been a highly addressed problem in robotics and computer vision. Majority of the recent work in literature has focused on classifying pre-segmented video clips, and some progress has also been made on joint detection and recognition of actions in complex video sequences. These methods, however, are not designed for wireless camera networks where the sensors have limited internal processing and communication capabilities. In this paper we present an efficient system for the joint detection and recognition of human actions using a network of wireless smart cameras. The foundation of our work is based on Deformable Part Models (DPMs) for detecting objects in static images. We have extended this framework to the single-view and multi-view video setting to jointly detect and recognize actions. We call this the Deformable Keyframe Model (DKM) and tightly integrate it within a centralized video analysis system. In our system, feature extraction is locally performed on-board wireless smart cameras, and the classification is performed at a base station with higher processing power. Our analysis demonstrates that this decoupling of the the recognition pipeline can significantly minimize the power and bandwidth consumed by the wireless cameras. We experimentally validate our DKMs on two data sets. We first demonstrate the competitiveness of our algorithm by comparing its performance against other state-of-the-art methods, on a publicly available dataset. Then, we extensively validate our system on a novel dataset called the Bosch Multiview Complex Action (BMCA) dataset. Our dataset consists of 11 actions continuously performed by 20 different subjects while being captured by cameras located at 4 different vantage points. In our experiments, we demonstrate that the presence of multiple-views improves the performance of action detection and recognition by about 15% over the single-view setting.