Model ensemble techniques often increase task performance in neural networks; however, they require increased time, memory, and management effort.In this study, we propose a novel method that replicates the effects of a model ensemble with a single model.Our approach creates K-virtual models within a single parameter space using K-distinct pseudotags and K-distinct vectors.Experiments on text classification and sequence labeling tasks on several datasets demonstrate that our method emulates or outperforms a traditional model ensemble with 1/K-times fewer parameters.