The deployment of machine learning in real-life computer vision applications is often subject to the degraded input data by various noise sources and time-varying conditions. Continual re-training is one potential solution to adapt to the real-time noise sources and achieve the target performance. However, this approach requires massive computing complexity for the training, particularly demanding on resource-constrained edge devices. To address this challenge, for the first time, we present a hardware-efficient framework named LEAF, tailored for the adaptation to degraded images (ADI) scenario. Leveraging both qualitative and quantitative profiling of CNN dynamics when operating on degraded images, we propose two techniques: 1) selective experience replay (SER)-based unimportant image skipping to reduce both the forward pass (FwP) and backward pass (BwP) costs, and 2) pseudo noise dithering (PND)-assisted extremely low precision quantization (3/4-bit) for error gradients to enable nearly full integer computing. Extensive experiments on CIFAR10 and Tiny ImageNet datasets employing VGGNet and ResNet, along with five different types of image degradations with varying strengths demonstrate virtually no accuracy degradation (< 0.5%) at 3/4-bits with 3.64 ~ 8.40× reduced forward and backward processing.