We propose a novel human-computer interface for video gaming systems. Unlike current systems that use active sensors (e.g. Kinect), in the proposed approach, the human-poses are captured by a single camera and used to control the course of the game. This eliminates the need for traditional hand-held controllers and allows for the player to be immersed into the virtual environment of the game. The success of the approach is achieve through a robust extraction of human silhouettes and by a simple but effective human-pose recognizer. The system employs an adaptive framework for background subtraction using Adaptive Local Principal Component Analysis (ALPCA), which is able to handle sudden as well as gradual changes in background. Next, the extracted human silhouettes are used to train a appearance-based algorithm that can recognize the human poses. The excellent results reported here justify the elimination of active sensors and game controllers for many traditional games.