A moving window scheme for detecting lanes and obstacles from the images captured by a CCD camera in an automobile is proposed. Processing the input dynamic images in real time requires high performance hardware as well as efficient software. The proposed moving window scheme relieves the requirements for detecting lanes and obstacles from the images in real time. The size of the moving window is optimally determined based upon road and automobile conditions to be big enough to detect the lanes and obstacles and to be small enough to process the detection procedure in real time. For each image frame, the moving window is newly defined and it is moved in a certain direction that is predicted by the Kalman filtering technique. By detecting the left and right lane marks of the driving lane, it becomes possible to search for obstacles within the lane. The obstacle can be verified through the correlation between the stochastic characteristics of the suspected obstacle and the actual obstacle in the database. The feasibility of the proposed algorithm is demonstrated through the simulated experiments of freeway driving.