Early hospital admission prediction at the triage stage is an important and challenging task for emergency departments (EDs), aimed at effectively managing and utilizing limited medical resources for critical patients. A retrospective study was conducted at MacKay Memorial Hospital (MMH) from 2011 to 2018, including 1,061,760 records of valid patients, using logistic regression (LR), eXtreme Gradient Boosting (XGBoost), Word2Vec, and bidirectional encoder representations from transformers (BERT). The chief complaints (CCs) and limited structured variables collected at triage are considered predictor variables. The results show that XGBoost achieves better prediction than LR with patient structured variables and better prediction than Word2Vec with patient CCs in terms of AUC and F-measure. We further propose the novel concept of generating expanded CCs as BERT input by integrating the original CCs with selected structured variables using XGBoost to predict the probability of patient admission. Among the structured variables, triage category, mode of arrival, age, arrival time, and fever status are the most important. This study demonstrates BERT's (in particular, BERT-ROS with 5 variables) superior prediction capability compared to other models by considering only patient CCs or expanded CCs in terms of AUC and F-measure. Moreover, given the low admission rates in Taiwan's EDs, this study employs imbalanced data processing to show that the proposed method enhances the predictive capability of hospitalization. These experimental results provide a reference model with associated variables for developing a hospital admission tool at triage, identifying the risk of stratification of critical patients.