Most previous Web-page summarization methods treat a Web page as plain text. However, such methods fail to uncover the full knowledge associated with a Web page needed in building a high-quality summary, because many of these methods do not consider the hidden relationships in the Web. Uncovering the hidden knowledge is important in building good Web-page summarizers. In this paper, we extract the extra knowledge from the clickthrough data of a Web search engine to improve Web-page summarization. Wefirst analyze the feasibility in utilizing the clickthrough data to enhance Web-page summarization and then propose two adapted summarization methods that take advantage of the relationships discovered from the clickthrough data. For those pages that are not covered by the clickthrough data, we design a thematic lexicon approach to generate implicit knowledge for them. Our methods are evaluated on a dataset consisting of manually annotated pages as well as a large dataset that is crawled from the Open Directory Project website. The experimental results indicate that significant improvements can be achieved through our proposed summarizer as compared to the summarizers that do not use the clickthrough data.

As the competition of Web search market increases, there is a high demand for personalized Web search to conduct retrieval incorporating Web users' information needs. This paper focuses on utilizing clickthrough data to improve Web search. Since millions of searches are conducted everyday, a search engine accumulates a large volume of clickthrough data, which records who submits queries and which pages he/she clicks on. The clickthrough data is highly sparse and contains different types of objects (user, query and Web page), and the relationships among these objects are also very complicated. By performing analysis on these data, we attempt to discover Web users' interests and the patterns that users locate information. In this paper, a novel approach CubeSVD is proposed to improve Web search. The clickthrough data is represented by a 3-order tensor, on which we perform 3-mode analysis using the higher-order singular value decomposition technique to automatically capture the latent factors that govern the relations among these multi-type objects: users, queries and Web pages. A tensor reconstructed based on the CubeSVD analysis reflects both the observed interactions among these objects and the implicit associations among them. Therefore, Web search activities can be carried out based on CubeSVD analysis. Experimental evaluations using a real-world data set collected from an MSN search engine show that CubeSVD achieves encouraging search results in comparison with some standard methods.

Latent semantic indexing (LSI) is a successful technology in information retrieval (IR) which attempts to explore the latent semantics implied by a query or a document through representing them in a dimension-reduced space. However, LSI is not optimal for document categorization tasks because it aims to find the most representative features for document representation rather than the most discriminative ones. In this paper, we propose supervised LSI (SLSI) which selects the most discriminative basis vectors using the training data iteratively. The extracted vectors are then used to project the documents into a reduced dimensional space for better classification. Experimental evaluations show that the SLSI approach leads to dramatic dimension reduction while achieving good classification results.

Web-page classification is much more difficult than pure-text classification due to a large variety of noisy information embedded in Web pages. In this paper, we propose a new Web-page classification algorithm based on Web summarization for improving the accuracy. We first give empirical evidence that ideal Web-page summaries generated by human editors can indeed improve the performance of Web-page classification algorithms. We then propose a new Web summarization-based classification algorithm and evaluate it along with several other state-of-the-art text summarization algorithms on the LookSmart Web directory. Experimental results show that our proposed summarization-based classification algorithm achieves an approximately 8.8% improvement as compared to pure-text-based classification algorithm. We further introduce an ensemble classifier using the improved summarization algorithm and show that it achieves about 12.9% improvement over pure-text based methods.

Most of current researches on Web page classification focus on leveraging heterogeneous features such as plain text, hyperlinks and anchor texts in an effective and efficient way. Composite kernel method is one topic of interest among them. It first selects a bunch of initial kernels, each of which is determined separately by a certain type of features. Then a classifier is trained based on a linear combination of these kernels. In this paper, we propose an effective way to optimize the linear combination of kernels. We proved that this problem is equivalent to solving a generalized eigenvalue problem. And the weight vector of the kernels is the eigenvector associated with the largest eigen-value. A support vector machine (SVM) classifier is then trained based on this optimized combination of kernels. Our experiment on the WebKB dataset has shown the effectiveness of our proposed method.

Currently, there are few efficient methods in practice for learning Bayesian networks from incomplete data, which affects their use in real world data mining applications. We present a general-duty method that estimates the (conditional) mutual information directly from incomplete datasets, EMI. EMI starts by computing the interval estimates of a joint probability of a variable set, which are obtained from the possible completions of the incomplete dataset. And then computes a point estimate via a convex combination of the extreme points, with weights depending on the assumed pattern of missing data. Finally, based on these point estimates, EMI gets the estimated (conditional) mutual information. We also apply EMI to the dependency analysis based learning algorithm by J. Cheng so as to efficiently learn BNs with incomplete data. The experimental results on Asia and Alarm networks show that EMI based algorithm is much more efficient than two search & scoring based algorithms, SEM and EM-EA algorithms. In terms of accuracy, EMI based algorithm is more accurate than SEM algorithm, and comparable with EM-EA algorithm.

For simplicity of calculation, dynamic Bayesian networks (DBNs) make assumptions that their evolvement follows Markov process and the transition probabilities in the evolvement are time-invariant. While this is not the case in many real complex systems. For the purpose of modeling these complex systems with DBNs, we attempt to add hidden variables to the evolutional process so as to build Markov models and expand the EM-EA algorithm to the DBNs learning in the presence of hidden variables. Moreover, as for the time-variant transition probabilities, we estimate the suffcient statistics of posterior time slices using polynomial fitting algorithm, and then learns the time-variant transition probabilities with both current sufficient statistics and estimated sufficient statistics. The theoretical analysis demonstrates the validity of the methods. The future work is to do experimental analysis with real complex systems.

Similarity measure of time series is an important subroutine in many KDD applications. Previous similarity models mainly focus on the prominent series behaviors by considering the whole information of time series. In this paper, we address the problem: which portion of information is more suitable for similarity measure for the data collected from a certain field. We propose a model for the retrieval and representation of the partial information in time series data, and a methodology for evaluating the similarity measurements based on partial information. The methodology is to retrieve various portions of information from the raw data and represent it in a concise form, then cluster the time series using the partial information and evaluate the similarity measurements through comparing the results with a standard classification. Experiments on data set from stock market give some interesting observations and justify the usefulness of our approach.

WebME is a Web mining environment (system) developed by Tsinghua University. It integrates many functions including: customization-based gathering, rough and subtle filtering, storing, indexing, recognizing, schema and data extracting, classifying, clustering and summarizing of Web pages, intelligent search engine, Web navigation, recommendation of Web information based on association of concept etc. We aim to utilize it as an experimental platform of Web mining on which we can design, implement, test and evaluate various algorithms of Web mining. In the paper, its architecture, main function and running environment are described.

An incremental method for learning Bayesian networks based on evolutionary computing, IEMA, is put forward. IEMA introduces the evolutionary algorithm and EM algorithm into the process of incremental learning; it can avoid getting into local maxima, and also incrementally learn Bayesian networks with high accuracy in the presence of missing values and hidden variables. In addition, we improved the incremental learning process by N. Friedman and M. Goldschmidt (1997). The experimental results verified the validity of IEMA. In terms of storage cost, IEMA is comparable with the incremental learning method of Friedman et al, while it is more accurate.

Classification, which involves finding rules that partition a given dataset into disjoint groups, is one class of data mining problems. Approaches proposed so far for mining classification rules from databases are mainly decision tree based on symbolic learning methods. In this paper, we combine artificial neural network and genetic algorithm to mine classification rules. Some experiments have demonstrated that our method generates rules of better performance than the decision tree approach and the number of extracted rules is fewer than that of C4.5.