As the elderly population grows, falling accidents become more frequent, and the need for fall-risk monitoring systems increases. Deep learning models for fall-risk movement detection neglect the connections between the terrain and fall-hazard movements. This issue can result in false alarms, particularly when a person encounters changing terrain. This work introduces a novel multisensor system that integrates terrain perception sensors with an inertial measurement unit (IMU) to monitor fall-risk on diverse terrains. Additionally, a dual-task learning (DTL) architecture that is based on a modified CNNLSTM model is implemented; it is used to determine fall-risk level and the terrain from sensor signals. Three fall-risk levels - "normal," "near-fall," and "fall" - are identified as being associated with "flat ground," "stepping up," and "stepping down" terrains. Ten young subjects performed 16 activities on flat and stepping terrains in a laboratory setting, and ten elderly individuals were recruited to perform four activities in the hospital. The accuracies of classification of fall-risk levels and terrains by the proposed system are 97.6% and 95.2%, respectively. The system detects pre-impact fall movements, with a fall prediction accuracy of 97.7% and an average lead time of 329ms for fall trials, revealing the model's effectiveness. The overall monitoring accuracy for elderly individuals is 99.8%, confirming the robustness of the proposed system. This work discusses the impact of sensor type and the model architecture of DTL on the classification of fall-risk levels across various terrains. The results demonstrate that the proposed method is reliable for monitoring the risk of falling.