OBJECTIVE: Coronavirus disease 2019 (COVID-19) has caused considerable morbidity and mortality, especially in patients with underlying health conditions. A precise prognostic tool to identify poor outcomes among such cases is desperately needed. METHODS: Total 400 COVID-19 patients with underlying health conditions were retrospectively recruited from 4 centers, including 54 dead cases (labeled as poor outcomes) and 346 patients discharged or hospitalized for at least 7 days since initial CT scan. Patients were allocated to a training set (n = 271), a test set (n = 68), and an external test set (n = 61). We proposed an initial CT-derived hybrid model by combining a 3D-ResNet10 based deep learning model and a quantitative 3D radiomics model to predict the probability of COVID-19 patients reaching poor outcome. The model performance was assessed by area under the receiver operating characteristic curve (AUC), survival analysis, and subgroup analysis. RESULTS: The hybrid model achieved AUCs of 0.876 (95% confidence interval: 0.752-0.999) and 0.864 (0.766-0.962) in test and external test sets, outperforming other models. The survival analysis verified the hybrid model as a significant risk factor for mortality (hazard ratio, 2.049 [1.462-2.871], P < 0.001) that could well stratify patients into high-risk and low-risk of reaching poor outcomes (P < 0.001). CONCLUSION: The hybrid model that combined deep learning and radiomics could accurately identify poor outcomes in COVID-19 patients with underlying health conditions from initial CT scans. The great risk stratification ability could help alert risk of death and allow for timely surveillance plans.