Linear regression is one of the most important and widely used techniques for data analysis. However, sometimes people are not satisfied with it because of the following two limitations: 1) its results are sensitive to outliers, so when the error terms are not normally distributed, especially when they have heavy-tailed distributions, linear regression often works badly; 2) its estimated coefficients tend to have high variance, although their bias is low. To reduce the influence of outliers, robust regression models were developed. Least absolute deviation (LAD) regression is one of them. LAD minimizes the mean absolute errors, instead of mean squared errors, so its results are more robust. To address the second limitation, shrinkage methods were proposed, which add a penalty on the size of the coefficients. The LASSO is one of these methods and it uses the L1-norm penalty, which not only reduces the prediction error and the variance of estimated coefficients, but also provides an automatic feature selection function. In this paper, we propose the regularized least absolute deviation (RLAD) regression model, which combines the nice features of the LAD and the LASSO together. The RLAD is a regularization method, whose objective function has the form of "loss + penalty." The "loss" is the sum of the absolute deviations and the "penalty" is the L1-norm of the coefficient vector. Furthermore, to facilitate parameter tuning, we develop an efficient algorithm which can solve the entire regularization path in one pass. Simulations with various settings are performed to demonstrate its performance. Finally, we apply the algorithm to solve the image reconstruction problem and find interesting results.

Recently, discussion has begun to consider serving the base of the economic pyramid for business and societal benefit. Suggestions for so doing have focused on business model innovation, improving internal capabilities, promoting stronger governance, and providing leapfrog technologies (such as ICT) to promote effective change. We consider the questions of what types of innovation are occurring when businesses and other organizations enter the base of the pyramid and what types of social, economic, and consumer benefits they bring to local communities. We base our findings on an analysis of more than 1,000 cases describing base of the pyramid activities

We introduce an emergent, collaborative filing system. In such a system, an individual is allowed to organize a subset of documents in a repository into a personal hierarchy and share the hierarchy with others. The system generates a "consensus" hierarchy from all users' personal hierarchies, which provides a full, common, and emergent view of all documents. We believe that collaborative filing helps translate personal, tacit knowledge into sharable structures, which help the user as well a community of which he or she is a part. Our filing system is suitable for any documents from text to multimedia files. Initial results on an experimental website show promise. For a knowledge task involving extensive document retrieval, hierarchies are not only used frequently but are also effective in identifying high quality documents. One surprising finding is how often subjects use others' personal hierarchies, and upon close examination, social networks play a key role as well.

The Internet and corporate intranets have brought a lot of information. People usually resort to search engines to find required information. However, these systems tend to use only one fixed ranking strategy regardless of the contexts. This poses serious performance problems when characteristics of different users, queries, and text collections are taken into account. We argue that the ranking strategy should be context specific and we propose a , new systematic method that can automatically generate ranking strategies for different contexts based on genetic programming (GP). The new method was tested on TREC data and the results are very promising.

Previous research has shown that researchers can generate medical hypotheses by using computers to analyze several, seemingly unrelated, medical literatures. In this work we suggest broader application for the ideas of literature-based discovery. Specifically, we suggest that literature-based discovery can be fruitful in areas other than medicine; that in addition to finding "cures" for "problems," literature-based discovery offers the possibility of finding new problems for existing technologies; that the analysis of a single literature may be sufficient for literature-based discovery; and that literature-based discovery can support individuals seeking to draw together ideas from various areas of inquiry, even if such connections have been previously made by others.We describe literature-based discovery experiments conducted on the World Wide Web that support these ideas.

We report experiments that use lexical statistics, such as word frequency counts, to discover hidden connections in the medical literature. Hidden connections are those that are unlikely to be found by examination of bibliographic citations or the use of standard indexing methods and yet establish a relationship between topics that might profitably be explored by scientific research. Our experiments were conducted with the MEDLINE medical literature database and follow and extend the work of Swanson.

Latent semantic indexing (LSI) is a statistical technique for improving information retrieval effectiveness. Here, we use LSI to assist in literature-based discoveries. The idea behind literature-based discoveries is that different authors have already published certain underlying scientific ideas that, when taken together, can be connected to hypothesize a new discovery, and that these connections can be made by exploring the scientific literature. We explore latent semantic indexing's effectiveness on two discovery processes: uncovering “nearby” relationships that are necessary to initiate the literature based discovery process; and discovering more distant relationships that may genuinely generate new discovery hypotheses. © 1998 John Wiley & Sons, Inc.

Don R. Swanson has undertaken a program of research to use the published medical literature as a source of discoveries. We have attempted to replicate his discovery of a connection between Raynaud's disease and dietary fish oil, as well as develop computer-based searching methods that could usefully support literature-based discoveries. We have been successful in replicating Swanson's discovery and have developed a method of discovery support based on the complete text of MEDLINE records. From these, we compute statistics based both on the frequency of tokens within a literature and on the number of records containing various tokens. We discuss the use of these statistics, suggesting that token and record frequencies are good indicators of literatures profitably related to some source literature, and that relative record frequencies are useful in isolating literatures with the potential of containing a discovery. © 1996 John Wiley & Sons, Inc.

The probability ranking principle retrieves documents in decreasing order of their predictive probabilities of relevance. Gordon and Lenk (1991) demonstrated that this principal is optimal within a signal detection—decision theory framework, and it maximizes the inquirer's expected utility for relevant documents. These results hold under three conditions: calibration, independent assessment of relevance by the inquirer, and certainty about the computed probabilities of relevance. We demonstrate that the probability ranking principle can be suboptimal with respect to expected utility when one of these conditions fails to hold. © 1992 John Wiley & Sons, Inc.

Information retrieval systems have used clustering of documents and queries to improve both retrieval efficiency and retrieval effectiveness. Normally, clustering involves grouping together static descriptions of documents by their similarity to each other, though user-based clustering suggests that usage patterns concerning co-relevance can form a basis for clustering. This article reports that clusters of co-relevant documents obtain increasingly similar descriptions when a genetic algorithm is used to adapt subject descriptions so that documents become more effective in matching relevant queries and failing to match nonrelevant queries. As a result of the increased similarity, clustering algorithms can more accurately group documents into useful clusters. The findings of this work were reached through simulation experiments. © 1991 John Wiley & Sons, Inc.

We challenge the probability ranking principle in information retrieval from the perspectives of (1) signal detection-decision theory and (2) utility theory. If three conditions are not met by an IR system that is producing predictive probabilities of relevance, then inquirers may incur costs that are too great by selecting first those documents that the system predicts have the highest probabilities of relevance. These three conditions are that predictive probabilities are well calibrated (predictively accurate); that they are reported with certainty; and that an inquirer independently assesses the relevance of all documents he or she retrieves. When these conditions are met, signal detection analysis with fixed decision-theoretic costs shows that the probability ranking principle is advisable. More generally, retrieval in adherence with the probability ranking principle is also advisable even when utility-theoretic costs (or benefits) that vary with the number of relevant documents retrieved are associated with retrieval. Specifically, we prove that the utility an inquirer receives from the relevant documents he or she retrieves is maximized by selecting those documents with the largest predictive probabilities of relevance. © 1991 John Wiley & Sons, Inc.

Simulation is a widely used investigative tool for exploring proposed systems without incurring the costs of actually building them. Studying the effectiveness of document retrieval systems by means of simulation, however, has remained elusive because of the guesswork involved in deciding whether a document with a given (simulated) description will be relevant to the information need of an inquirer expressed by a given (simulated) query. In this article, a simulation method is described for estimating recall and fallout in a document retrieval system. Examples are presented to illustrate the method, and it is justified probabilistically. By using this method, we may compare the effectiveness of employing various indexing procedures, compare the effectiveness of employing various matching functions, or obtain absolute effectiveness measures for a proposed system. © 1990 John Wiley & Sons, Inc.

Abstract Considerable interest has been shown in problems associated with ranking journals in accord with their relative importance. With the publication of Journal Citation Reports, regularly updated ranking indicators, based on citation analysis, are made readily available. These indicators have been shown to have significant correlations with scientists' evaluations of the relative importance of journals in the biomedical sciences, but not the physical sciences. This article examines the situation in the social sciences, and finds a strong association between sociologists' evaluations of journal rank and particular citation rank indicators. This suggests that Social Science Journal Citation Reports could prove a valuable aid to journal selection for libraries and information systems.

Abstract The problems of scientists working in “less developed countries” (LDCs) have recently attracted considerable attention, as has their low level of scientific output This article presents an examination of the editorial evaluation of papers which LDC authors are able to produce, in an attempt to illuminate their problems through analysis of their output. LDC authors are shown to experience far higher rejection rates than authors in developed countries (57% vs. 17%) and to be faulted on specific points which tend to indicate that their major problems are those of maintaining current awareness. It is thus indicated that the problems LDC researchers experience in accessing scientific information are a major factor in inhibiting the quality and quantity of their own scientific information output.