Web ad effect evaluation is a challenging problem in web marketing research. Although the analysis of web ad effectiveness has achieved excellent results, there are still some deficiencies. First, there is a lack of an in‐depth study of the relevance between advertisements and web content. Second, there is not a thorough analysis of the impacts of users and advertising features on user browsing behaviors. And last, the evaluation index of the web advertisement effect is not adequate. Given the above problems, we conducted our work by studying the observer's behavioral pattern based on multimodal features. First, we analyze the correlation between ads and links with different searching results and further assess the influence of relevance on the observer's attention to web ads using eye‐movement features. Then we investigate the user's behavioral sequence and propose the directional frequent‐browsing pattern algorithm for mining the user's most commonly used browsing patterns. Finally, we offer the novel use of “memory” as a new measure of advertising effectiveness and further build an advertising memory model with integrated multimodal features for predicting the efficacy of web ads. A large number of experiments have proved the superiority of our method.