This paper presents an algorithm for background modeling and foreground detection that uses scaling coefficients, which are defined with a new color model called lightness-red-green-blue (LRGB). They are employed to compare two images by finding pixels with scaled lightness. Three backgrounds are used: 1) verified background with pixels that are considered as background; 2) testing background with pixels that are tested several times to check if they belong to the background; and 3) final background that is a combination of the testing and verified background (the testing background is used in places, where the verified background is not defined). If a testing background pixel matches pixels from previous frames (the match is tested using scaling coefficients), it is copied to the verified background, otherwise the pixel is set as the weighted average of the corresponding pixels of the last input images. After the background is computed, foreground objects are detected by using the scaling coefficients and additional criteria. The algorithm was evaluated using the SABS data set, Wallflower data set and a subset of the CDnet 2014 data set. The average F measure and sensitivity with the SABS Data set were 0.7109 and 0.8725, respectively. In the Wallflower data set, the total number of errors was 5280 and the total F-measure was 0.9089. In the CDnet 2014 data set, the F-measure for the baseline test case was 0.8887 and for the shadow test case was 0.8300.