Continual test-time domain adaptation (CTTA) aims to adapt a pre-trained source model to a stream of continually evolving unlabeled target domains, facilitating model deployment in dynamic and non-stationary environments. Contemporary works usually encode domain-specific (DS) style information in a domain-agnostic manner, synchronizing with the learning of domain-invariant (DI) semantic information. This scheme forces DS information to be optimized using the weights of the previous domain, corrupted by cross-domain discrepancies, and hence leads to error accumulation and catastrophic forgetting issues. Inspired by the Attribute Memory Model (AMM) in brain neuroscience, we propose a dual domain-attribute learning framework based on independent asynchronous updates, aiming to imitate how brain learns new knowledge without forgetting. Concretely, we explicitly decompose the continual adaptation process into two complementary systems: an event-based learning system (ELS) that captures DS style representations and a knowledge-based learning system (KLS) that concentrates on the DI structural characteristics. The ELS first detects differences in the distribution of data streams, and actively builds an adapter pool for new latent domains. The KLS adopts a cross-domain shared adapter emphasizing general knowledge, and cooperates with the adapter from ELS to jointly guide adaptation. To make DS and DI knowledge collaboratively working, we exploit a gradient conflict solver to ease the conflict between the past and current DI knowledge, realizing a win-win game (i.e., no interference adaptation) across evolving domains. Our framework have been extensively evaluated on four benchmarks and outperformed the state-of-the-art approaches on both segmentation and classification CTTA tasks.