Scale analysis plays a vital role in pedestrian detection. Conventional approaches usually directly concatenate multi-scale outputs, which is only capable of modeling first-order dependency among various scales. In contrast, this work proposes a novel scale-context modeling scheme by exploiting the highly nonlinear dependency among scales. The proposed scheme aggregates output response maps from mid-results of convolutional layers via a bi-directional recurrent sub-network. Therefore scale information could flow among different layers and implicit underlying dependency structure information in the scale space would be disclosed, which yields more consistency detection. Experimental results on Caltech Pedestrian detection benchmark demonstrate the superior detection (state-of-the-art miss rate of 8.56%) of the proposed method over prior art.