With the rapid development of wearable cameras, it is now feasible to considerably increase the collection of egocentric video for first-person visual perception. However, the development is hindered by a shortage of multi-modal egocentric activity datasets. Furthermore, the catastrophic forgetting problem of multimodal continual activity learning, as a branch of continual learning, has not been thoroughly explored, which makes accumulating a larger collection of multi-modal activity data more urgent. To address this shortage, we propose a multi-modal egocentric activity dataset for continual activity learning named UESTC-MMEA-CL in this paper. The dataset is collected using our self-developed glasses with a first-person camera and wearable sensors, and it contains synchronized data of video, accelerometers, and gyroscopes for 32 types of daily activities performed by 10 participants who wore our glasses. Statistical analysis of the sensor data is given to show the auxiliary effects of activity recognition. We report the results of egocentric activity recognition of three modalities (RGB, acceleration, and gyroscope) separately and jointly on a base network architecture. We thoroughly evaluated four baseline methods with different multimodal combinations to explore the catastrophic forgetting in continual learning on UESTC-MMEA-CL. We hope that the UESTC-MMEA-CL dataset can act as a facilitator for future studies on continual learning for first-person activity recognition in wearable applications. You can download preliminary data fromhttps://ivipclab.github.io/publication_uestc-mmea-cl/mmea-cl. The data is currently used to solve the problems of multimodal continual learning of activities.