In this paper, authentication for mobile radio frequency identification (RFID) systems with low-cost tags is investigated. To this end, an adaptive modulus (AM) encryption algorithm is first proposed. To further enhance security without requiring additional storage for new key matrices, a self-updating encryption order (SUEO) algorithm is designed. Furthermore, a diagonal block local transpose key matrix (DBLTKM) encryption algorithm is presented, which effectively expands the feasible domain of the key space. Building upon these three algorithms, a novel joint AM-SUEO-DBLTKM encryption algorithm is constructed. Making full use of the strengths of the proposed joint algorithm, a two-way RFID authentication protocol, named AM-SUEO-DBLTKM-RFID, is proposed specifically for mobile RFID systems. In addition, the Burrows-Abadi-Needham (BAN) logic and security analysis indicate that the proposed AM-SUEO-DBLTKM-RFID protocol can effectively combat various typical attacks. Numerical results demonstrate that the proposed AM-SUEO-DBLTKM algorithm can save 99.59% of tag storage over traditional algorithms. Finally, the proposed AM-SUEO-DBLTKM-RFID protocol achieves both low computational complexity and low storage overhead, making it well-suited for deployment in resource-constrained, low-cost RFID tags.