With the growing awareness of sustainability concerns in many application domains, energy-harvesting (EH) devices are increasingly being used with traditional non-energy-harvesting (non-EH) devices. This paper proposes a reinforcement learning (RL)-based task offloading and scheduling strategy called DTORA for a hybrid EH and non-EH enabled multi-access edge computing environment where EH devices harvest energy from solar radiation. The applications running on user devices can have varying levels of criticality (mixed-criticality). We formulate a mixed-integer energy and latency minimization programming problem based on the system and application model. To solve this, we use a recurrent neural network based long short-term memory (LSTM) model for solar energy prediction, and a Double Deep Q-learning is used for task-offloading decisions. The proposed strategy (DTORA) is benchmarked against several state-of-the-art (SOA) strategies and other baseline approaches, including SCOPE, OCO (Offloading Cost Optimization), a hybrid Particle Swarm Optimization and Genetic Algorithm (PSOGA), and Selective-Greedy (SG). The proposed strategy outperforms these strategies in terms of latency, energy consumption of user devices, task failure rate, and critical task failures by 39%, 77.51%, 60.98%, and 69.94%, respectively (on average). Compared to the existing best-performing strategy, DTORA achieves improvements of 17.45% for latency, 58.54% for energy consumption, 23.34% for task failure rate, and 27.77% for critical task failures. This improvement can be attributed to improved edge-cloud cooperation, an efficient energy prediction model, and efficient RL-based task offloading and scheduling in our proposed strategy.