In this paper we introduce a gesture-based robot control framework, we discuss the adopted design principles and we report results about its evaluation with humans. Gesture-based control using wearable devices may constitute a novel form of human-robot interaction, but its implications have not been discussed in the literature. We discuss the main challenging issues, possible design guidelines and an open source, freely available implementation using commercially available devices and robots. The overall performance of the architecture, as well as its validation with 27 untrained volunteers, is reported.