Federated learning (FL) enables multiple clients to collaboratively train models without exposing raw data. However, most FL research assumes that all clients have fully labeled data, which is impractical for many real-world applications. To this end, we focus on semi-supervised FL (SSFL), where data samples of each client are partially labeled. Existing SSFL methods often ignore three inherent characteristics of FL: limited communication resources, data heterogeneity and system heterogeneity, which severely hinder convergence stability and efficiency. This paper proposes E-FedAC, a novel SSFL framework that tackles these challenges through alternating client-to-client (C2C) collaboration. Specifically, E-FedAC employs a two-stage clustering strategy in each training round while concurrently considering differences in client computing/communication capabilities to mitigate the impact of intra-cluster stragglers. The first stage employs similarity clustering to group clients based on local data distribution, which allows similar clients to share knowledge and generate high-quality pseudo-labels for unlabeled data. In the second stage, clients are re-grouped using a dissimilarity clustering strategy to approximate the IID setting at the cluster level, thereby alleviating the bias induced by Non-IID data. E-FedAC utilizes a reinforcement learning algorithm to adaptively determine the average number of intra-cluster iterations across all clusters for both clustering stages, balancing between labeling assistance from similar clients and unbiased optimization from dissimilar clients. Additionally, E-FedAC dynamically optimizes local updating frequencies for each generated cluster based on this average and individual cluster capabilities, further reducing the waiting time across clusters. Extensive evaluations demonstrate that E-FedAC can provide up to 4.02× speedup and 61.74% savings in communication costs compared with existing benchmarks.