Code search refers to the retrieval of code segments that best match the developer's needs from a large code base, whose needs are generally expressed in natural language. So far, the biggest difficulty in code search is the semantic gap between code language and natural language when matching, which is very different in grammar and language structure. In this paper, we propose a novel deep learning framework CodeFuse to achieve effective and accurate source code search by constructing a new mechanism to learn the rich semantics of source code and natural language queries from the two modalities of text and structure. We conduct extensive experiments using Java and Python corpus in CodeSearchNet which is large-scale and multi-language. Our multimodal approach surpasses baselines at most by 24.70% in MRR score, and 14.40% in NDCG score. The results show that our model can focus on information that is strongly correlated between code and query features without redundancy and improve the performance of code search.