Sequential recommendation systems often struggle to accurately predict user preferences when limited to historical browsing data. We present a novel approach that combines recommendation systems with search engine methodologies, introducing a generative intent prediction model that leverages both item view histories and historical search queries. The model is enhanced by incorporating user interaction data from search engine result pages (SERP), leading to more accurate query predictions aligned with actual user behavior. By integrating this intent prediction model into sequential recommendation frameworks through a query expansion-inspired approach, we demonstrate significant performance improvements over traditional methods, particularly in challenging scenarios where conventional approaches fall short.

Abstract The ability to predict blunders in chess plays a crucial role in improving players’ performance and enabling strategic decision-making. We introduce a novel, scalable, and personalized blunder prediction model for chess. Unlike prior work requiring a separate model per player, our unified architecture learns a collaborative user embedding space, allowing it to generalize weaknesses across players and new users. Our hybrid model, inspired by Deep Factorization Machines (DeepFM), fuses a frozen pre-trained CNN (for board embeddings) with dynamically learned user embeddings to model player-board interactions while still utilizing metadata about the state of the board and the user. We demonstrate that this latent ’blunder profile’ is a significantly more powerful predictor of error than a player’s explicit Elo rating. The system achieves state-of-the-art performance (0.801 AUC) on both immediate and non-immediate blunders, offering an efficient and data-sparse-friendly solution for personalized chess analysis. Ultimately, this approach demonstrates the practical viability of deep personalization in complex strategy games, facilitating highly efficient, user-centric learning environments.

Promotional campaigns are essential drivers of customer engagement and revenue in e-commerce. Maintaining these campaigns within budget constraints requires targeted allocation, traditionally achieved through causal uplift models that rely on vast datasets of user interactions, including non-converted sessions, which introduce challenges such as noisy data, attribution complexity and imbalanced outcomes. We propose a novel approach using converted-only data, which reduces training data size, simplifies attribution, improves efficiency, and mitigates the impact of non-converted interactions. We present a generalized framework for budget constrained promotion allocation with converted-only data and validate it through a benchmarking study and multiple large-scale deployments at Booking.com, positively impacting the experience of millions of customers worldwide. Our results demonstrate that the proposed method is competitive with standard modeling approaches and, in some cases, significantly outperforms them.

Large Language Models (LLMs) are increasingly integrated into real-world applications, making equitable treatment of all demographic groups a critical concern. Existing fairness evaluations often rely on binary, template-based tests, which overlook subtle disparities in open-ended responses. We present DiFair-LLM, a model-agnostic framework for detecting and quantifying fairness disparities - any unequal treatment that benefits or disadvantages a demographic group. DiFair-LLM uses open-ended, group-specific and neutral prompts, measures semantic distances between groups’ responses, applies non-parametric statistical tests, and ranks groups by deviation from a neutral baseline. Evaluations across eight state-of-the-art LLMs and multiple demographic attributes reveal minimal disparities for gender but significant differences for age, especially older adults, and ethnicity, with the largest gaps affecting certain non-Caucasian groups. By mapping nuanced patterns of differential treatment rather than flagging only overt bias, DiFair-LLM offers a practical, reproducible approach for auditing fairness and guiding more inclusive LLM deployments.

Large language models (LLMs) often appear to excel on public benchmarks, but these high scores may mask an overreliance on datasetspecific surface cues rather than true language understanding.We introduce the Chameleon Benchmark Overfit Detector (C-BOD), a meta-evaluation framework designed to reveal such overfitting.C-BOD systematically rephrases benchmark inputs via a parameterized transformation that preserves semantic content and labels, enabling the detection of performance degradation indicative of superficial pattern reliance.We conduct extensive experiments across two datasets, three rephrasing models, and multiple distortion levels, evaluating 32 state-of-the-art LLMs.On the MMLU benchmark, C-BOD reveals an average performance drop of 2.75% under modest rephrasings, with over 80% of models exhibiting statistically significant differences.Notably, higherperforming models and larger LLMs tend to show greater sensitivity, suggesting a deeper dependence on benchmark-specific phrasing.Due to its dataset and model-agnostic design, C-BOD can be easily integrated into evaluation pipelines and offers a promising foundation for overfitting mitigation strategies.Our findings challenge the community to look beyond leaderboard scores and prioritize resilience and generalization in LLM evaluation.Our code and benchmark datasets are available at

As new products are emerging daily, recommendation systems are required to quickly adapt to possible new domains without needing extensive retraining. This work presents ''X-Cross'' -- a novel cross-domain sequential-recommendation model that recommends products in new domains by integrating several domain-specific language models; each model is fine-tuned with low-rank adapters (LoRA). Given a recommendation prompt, operating layer by layer, X-Cross dynamically refines the representation of each source language model by integrating knowledge from all other models. These refined representations are propagated from one layer to the next, leveraging the activations from each domain adapter to ensure domain-specific nuances are preserved while enabling adaptability across domains. Using Amazon datasets for sequential recommendation, X-Cross achieves performance comparable to a model that is fine-tuned with LoRA, while using only 25% of the additional parameters. In cross-domain tasks, such as adapting from Toys domain to Tools, Electronics or Sports, X-Cross demonstrates robust performance, while requiring about 50%-75% less fine-tuning data than LoRA to make fine-tuning effective. Furthermore, X-Cross achieves significant improvement in accuracy over alternative cross-domain baselines. Overall, X-Cross enables scalable and adaptive cross-domain recommendations, reducing computational overhead and providing an efficient solution for data-constrained environments.

Review personalization aims at presenting the most relevant reviews of a product according to the preferences of the individual user. Existing studies of review personalization use the reviews authored by the user as a proxy for their preferences, and henceforth as a means for learning and evaluating personalization quality. In this work, we suggest using review votes rather than authorship for personalization. We propose MAGLLM, an approach that leverages heterogeneous graphs for modeling the relationships among reviews, products, and users, with large language model (LLM) to enrich user representation on the graph. Our evaluation over a unique public dataset that includes user voting information indicates that the vote signal yields substantially higher personalization performance across a variety of recommendation methods and e-commerce domains. It also indicates that our graph-LLM approach outperforms comparative baselines and algorithmic alternatives. We conclude with concrete recommendations for e-commerce platforms seeking to enhance their review personalization experience.

Product titles play a central role in connecting e-commerce buyers and sellers, concisely conveying the product information required to facilitate a transaction. Product titles are prominently presented in e-commerce search results, recommendations, and browse pages. In addition, many e-commerce tasks, such as matching, categorization, and product recommendations, heavily rely on product titles as a signal. In this paper, we report a comprehensive analysis of the linguistic characteristics of e-commerce titles. Specifically, we consider syntax, content, order of words, and attribute distribution. We compare these characteristics to those of the language used in other e-commerce and web corpora. Our analysis reveals a variety of unique properties of e-commerce title language. We consider the practical implications of our analysis with an empirical evaluation of modeling approaches on two real-world e-commerce tasks directly involving titles. Our findings suggest a number of practical approaches to applying natural language processing techniques over e-commerce titles. Based on these findings, we developed a set of clear, actionable guidelines for creating effective titles that can be adapted by e-commerce platforms.

The increasing application of machine learning (ML) in critical areas such as healthcare and finance highlights the importance of fairness in ML models, challenged by biases in training data that can lead to discrimination. We introduce ‘FairUS’, a novel pre-processing method for reducing bias in ML models utilizing the Conditional Generative Adversarial Network (CTGAN) to synthesize upsampled data. Unlike traditional approaches that focus solely on balancing subgroup sample sizes, FairUS strategically optimizes the quantity of synthesized data. This optimization aims to achieve an ideal balance between enhancing fairness and maintaining the overall performance of the model. Extensive evaluations of our method over several canonical datasets show that the proposed method enhances fairness by 2.7 times more than the related work and 4 times more than the baseline without mitigation, while preserving the performance of the ML model. Moreover, less than a third of the amount of synthetic data was needed on average. Uniquely, the proposed method enables decision-makers to choose the working point between improved fairness and model’s performance according to their preferences.

This workshop is dedicated to emphasizing the pivotal role of temporal dynamics in advancing recommender systems across various fields. While the significance of temporal factors in user behavior is widely acknowledged, effectively integrating these aspects into recommendation algorithms remains a complex challenge. In this workshop, we aim to showcase the application of temporal aspects in recommender systems across diverse domains such as healthcare, e-commerce, fashion, banking, travel, and film. We believe that this workshop will contribute to advancing temporal methodologies in recommender systems, ultimately leading to more precise recommendations.

Predicting the success of marketing campaigns on social media can help improve campaign managers’ decision-making (e.g., deciding to stop a marketing campaign) and thus increase their profits. Most research in the field of online marketing has focused on analyzing users’ behavior rather than improving campaign manager decision-making. Furthermore, determining the success of marketing campaigns is quite challenging due to the large number of possible metrics that must be analyzed daily. In this study, we suggest a method that incorporates machine learning models with traditional business rules to provide daily decision recommendations, based on the various metrics and considerations, and aimed at achieving the campaign’s goals. We evaluate our approach on a unique dataset collected from the most popular social networks, Facebook and Instagram. Our evaluation demonstrates the proposed method’s ability to outperform an expert-based method and the machine learning baselines examined, and dramatically increase the campaign managers’ profits.

BACKGROUND: Pancreatic ductal adenocarcinoma (PDAC) remains a serious threat to health, with limited effective therapeutic options, especially due to advanced stage at diagnosis and its inherent resistance to chemotherapy, making it one of the leading causes of cancer-related deaths worldwide. The lack of clear treatment directions underscores the urgent need for innovative approaches to address and manage this deadly condition. In this research, we repurpose drugs with potential anti-cancer activity using machine learning (ML). METHODS: We tackle the problem by using a neural network trained on drug-target interaction information enriched with drug-drug interaction information, which has not been used for anti-cancer drug repurposing before. We focus on eravacycline, an antibacterial drug, which was selected and evaluated to assess its anti-cancer effects. RESULTS: Eravacycline significantly inhibited the proliferation and migration of BxPC-3 cells and induced apoptosis. CONCLUSION: Our study highlights the potential of drug repurposing for cancer treatment using ML. Eravacycline showed promising results in inhibiting cancer cell proliferation, migration and inducing apoptosis in PDAC. These findings demonstrate that our developed ML drug repurposing models can be applied to a wide range of new oncology therapeutics, to identify potential anti-cancer agents. This highlights the potential and presents a promising approach for identifying new therapeutic options.

The research area of recommender systems (RS) in e-commerce has become extremely popular in recent years. However, traditional RSs tend to recommend popular items, while niche (long-tail) items are often neglected, which is known as the long-tail problem. However, recent studies found that tail items are one of the key success factors in the e-commerce world. The availability of such items encompasses relatively high marginal profits and boosts the sales of popular short-head items. We suggest promoting long-tail items by leveraging the short-head items’ advantages and exposing the user to a tail item that may have not been considered otherwise. We use a classification model and statistical tools to generate personalized recommendations of a long-tail item considering a short-head item that has already been clicked. The uniqueness of our method lies in the combination of tail and head items to uplift the exposure of the latter and in using an applicable solution to deal with the extreme volume of tail items. We demonstrate the effectiveness of our method on real-world data from eBay and provide an analysis of the long-tail phenomenon and consumption behavior.

MOTIVATION: The process of drug discovery is notoriously complex, costing an average of 2.6 billion dollars and taking ∼13 years to bring a new drug to the market. The success rate for new drugs is alarmingly low (around 0.0001%), and severe adverse drug reactions (ADRs) frequently occur, some of which may even result in death. Early identification of potential ADRs is critical to improve the efficiency and safety of the drug development process. RESULTS: In this study, we employed pretrained large language models (LLMs) to predict the likelihood of a drug being withdrawn from the market due to safety concerns. Our method achieved an area under the curve (AUC) of over 0.75 through cross-database validation, outperforming classical machine learning models and graph-based models. Notably, our pretrained LLMs successfully identified over 50% drugs that were subsequently withdrawn, when predictions were made on a subset of drugs with inconsistent labeling between the training and test sets. AVAILABILITY AND IMPLEMENTATION: The code and datasets are available at https://github.com/eyalmazuz/DrugWithdrawn.

Usually, image- and radar-based data are used to perform environmental characteristics related tasks in autonomous cars, while the use of valuable sensor data from the Controller Area Network (CAN) bus has been limited. The vehicle’s CAN bus data consist of multivariate time series data, such as velocity, RPM, and acceleration, which contain meaningful information about the vehicle dynamics and environmental characteristics. The ability to use these data to sense the environment, along with the sight sense (from image-based data), can prevent a single point of failure when image- or radar-based data are missing or incomplete, and contribute to increased understanding of the vehicle’s environment. Moreover, a solution that does not rely on image- or radar-based data also addresses concerns about privacy and the use of location-based data. We present DeepCAN, a novel hybrid method for road type classification that utilizes solely vehicle dynamics data and combines two main approaches for time series classification. In the end-to-end approach, a long short-term memory autoencoder (LSTM AE) is trained, and the learned embedding serves as the input to a fully convolutional network autoencoder (FCN AE), while the feature-based approach utilizes an XGBoost classifier with aggregated time series feature representation. In our comprehensive evaluation on two real-world datasets, we assessed the performance of each model component as an independent solution, as well as a model integrating all of the components in a hybrid solution. The results demonstrate DeepCAN’s efficiency and accuracy and provide a solid basis for its future use by the automobile industry.

Basketball is one of the most popular types of sports in the world. Recent technological developments have made it possible to collect large amounts of data on the game, analyze it, and discover new insights. We propose a novel approach for modeling basketball games using deep reinforcement learning. By analyzing multiple aspects of both the players and the game, we are able to model the latent connections among players' movements, actions, and performance, into a single measure - the Q-Ball. Using Q-Ball, we are able to assign scores to the performance of both players and whole teams. Our approach has multiple practical applications, including evaluating and improving players' game decisions and producing tactical recommendations. We train and evaluate our approach on a large dataset of National Basketball Association games, and show that the Q-Ball is capable of accurately assessing the performance of players and teams. Furthermore, we show that Q-Ball is highly effective in recommending alternatives to players' actions.

A bundle is a pre-defined set of items that are collected together. In many domains, bundling is one of the most important marketing strategies for item promotion, commonly used in e-commerce. Bundle recommendation resembles the item recommendation task, where bundles are the recommended unit, but it poses additional challenges; while item recommendation requires only user and item understanding, bundle recommendation also requires modeling the connections between the various items in a bundle. Transformers have driven the state-of-the-art methods for set and sequence modeling in various natural language processing and computer vision tasks, emphasizing the understanding that the neighbors of an element are of crucial importance. Under some required adjustments, we believe the same applies for items in bundles, and better capturing the relations of an item with other items in the bundle may lead to improved recommendations. To address that, we introduce BRUCE - a novel model for bundle recommendation, in which we adapt Transformers to represent data on users, items, and bundles. This allows exploiting the self-attention mechanism to model the following: latent relations between the items in a bundle; and users’ preferences toward each of the items in the bundle and toward the whole bundle. Moreover, we examine various architectures to integrate the items’ and the users’ information and provide insights on architecture selection based on data characteristics. Experiments conducted on three benchmark datasets show that the proposed approach contributes to the accuracy of the recommendation and substantially outperforms state-of-the-art methods

Collaborative filtering (CF) is a highly effective recommendation approach based on preference patterns observed in user-item interaction data. Since pure collaborative methods can have certain limitations, e.g., when the data is sparse, hybrid approaches are a common solution, as they are able to combine collaborative information with side-information (SI) about the items. In this work, we explore the value of subtitle information for the problem of movie recommendation. Differently from previously explored types of movie SI, e.g., titles or synopsis, subtitles are not only longer, but also contain unique information that may help us to predict more accurately if a user will enjoy a movie. To assess the usefulness of subtitles, we propose a technical framework named SubtitleCF that combines user and item embeddings derived from interaction data and SI. The subtitles may be embedded in different ways, e.g., Latent Dirichlet Allocation (LDA) and neural techniques. Computational experiments with a framework instantiation that relies on Bayesian Personalized Ranking (BPR) as industry-strength method for item ranking and different text embedding methods demonstrate the value of subtitles in terms of prediction accuracy and coverage. Moreover, a user study (N=247) reveals that the information contained in subtitles can be leveraged to improve the decision-making processes of users.

MOTIVATION: Teratogenic drugs can cause severe fetal malformation and therefore have critical impact on the health of the fetus, yet the teratogenic risks are unknown for most approved drugs. This article proposes an explainable machine learning model for classifying pregnancy drug safety based on multimodal data and suggests an orthogonal ensemble for modeling multimodal data. To train the proposed model, we created a set of labeled drugs by processing over 100 000 textual responses collected by a large teratology information service. Structured textual information is incorporated into the model by applying clustering analysis to textual features. RESULTS: We report an area under the receiver operating characteristic curve (AUC) of 0.891 using cross-validation and an AUC of 0.904 for cross-expert validation. Our findings suggest the safety of two drugs during pregnancy, Varenicline and Mebeverine, and suggest that Meloxicam, an NSAID, is of higher risk; according to existing data, the safety of these three drugs during pregnancy is unknown. We also present a web-based application that enables physicians to examine a specific drug and its risk factors. AVAILABILITY AND IMPLEMENTATION: The code and data is available from https://github.com/goolig/drug_safety_pregnancy_prediction.git. SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.

BACKGROUND: Drug-drug interactions (DDIs) are preventable causes of medical injuries and often result in doctor and emergency room visits. Previous research demonstrates the effectiveness of using matrix completion approaches based on known drug interactions to predict unknown Drug-drug interactions. However, in the case of a new drug, where there is limited or no knowledge regarding the drug's existing interactions, such an approach is unsuitable, and other drug's preferences can be used to accurately predict new Drug-drug interactions. METHODS: We propose adjacency biomedical text embedding (ABTE) to address this limitation by using a hybrid approach which combines known drugs' interactions and the drug's biomedical text embeddings to predict the DDIs of both new and well known drugs. RESULTS: Our evaluation demonstrates the superiority of this approach compared to recently published DDI prediction models and matrix factorization-based approaches. Furthermore, we compared the use of different text embedding methods in ABTE, and found that the concept embedding approach, which involves biomedical information in the embedding process, provides the highest performance for this task. Additionally, we demonstrate the effectiveness of leveraging biomedical text embedding for additional drugs' biomedical prediction task by presenting text embedding's contribution to a multi-modal pregnancy drug safety classification. CONCLUSION: Text and concept embeddings created by analyzing a domain-specific large-scale biomedical corpora can be used for predicting drug-related properties such as Drug-drug interactions and drug safety prediction. Prediction models based on the embeddings resulted in comparable results to hand-crafted features, however text embeddings do not require manual categorization or data collection and rely solely on the published literature.

Deep learning (DL) algorithms have played a major role in achieving state-of-the-art (SOTA) performance in various learning applications, including computer vision, natural language processing, and recommendation systems (RSs). However, these methods are based on a vast amount of data and do not perform as well when there is a limited amount of data available. Moreover, some of these applications (e.g., RSs) suffer from other issues such as data sparsity and the cold-start problem. While recent research on RSs used DL models based on side information (SI) (e.g., product reviews, film plots, etc.) to tackle these challenges, we propose boosting neural network (BNN), a new DL framework for capturing complex patterns, which requires just a limited amount of data. Unlike conventional boosting, BNN does not sum the predictions generated by its components. Instead, it uses these predictions as new SI features which enhances accuracy. Our framework can be utilized for many problems, including classification, regression, and ranking. In this paper, we demonstrate BNN's use for addressing a classification task. Comprehensive experiments conducted to illustrate BNN's effectiveness on three real-world datasets demonstrated its ability to outperform existing SOTA models for classification tasks (e.g., clickthrough rate prediction).

The identification of criminals' behavioral patterns can be helpful for solving crimes. Currently, in order to perform this task, police investigators manually extract criminals' behavioral patterns (also referred to as criminals' modus operandi) from a large corpus of police reports. These patterns are compared to the patterns observed in an ongoing criminal investigation to identify similarities that may link the suspect to other documented crimes. Due to the large number of historical cases, this manual process is time consuming, very costly in terms of police resources, and limits the investigators' ability to solve open cases. In this study, we propose an automatic and language independent method for extracting behavioral patterns from police reports. Relying on the extracted behavioral patterns as input, we utilize a Siamese neural network to identify burglaries committed by the same criminals. Experiments performed using a large dataset of police reports written in Hebrew provided by the Israel Police demonstrate the proposed method's high performance, achieving an AUC above 0.9. Using our method, we are also able to identify potential suspects for 22.41% of the open burglary cases in Israel.

Many sellers on e-commerce platforms offer buyers product bundles, which package together two or more different items. The identification of such bundles is a necessary step to support a variety of related services, from recommendation to dynamic pricing. In this work, we present a comprehensive study of bundle identification on a large e-commerce website. Our analysis of bundle compared to non-bundle listed items reveals several key differentiating characteristics, spanning the listing's title, image, and attributes. Following, we experiment with a multi-modal classifier, which takes advantage of these characteristics as features. Our analysis also shows that a bundle indicator input by sellers tends to be highly noisy and carries only a weak signal. The bundle identification task therefore faces the challenge of having a small set of manually-labeled clean examples and a larger set of noisy-labeled examples, in conjunction with class imbalance due to the relative scarcity of bundles.

The widespread use of machine learning algorithms and the high level of expertise required to utilize them have fuelled the demand for solutions that can be used by non-experts. One of the main challenges non-experts face in applying machine learning to new problems is algorithm selection - the identification of the algorithm(s) that will deliver top performance for a given dataset, task, and evaluation measure. We present AutoGRD, a novel meta-learning approach for algorithm recommendation. AutoGRD first represents datasets as graphs and then extracts their latent representation that is used to train a ranking meta-model capable of accurately recommending top-performing algorithms for previously unseen datasets. We evaluate our approach on 250 datasets and demonstrate its effectiveness both for classification and regression tasks. AutoGRD outperforms state-of-the-art meta-learning and Bayesian methods.

The task of attack attribution, i.e., identifying the entity responsible for an attack, is complicated and usually requires the involvement of an experienced security expert. Prior attempts to automate attack attribution apply various machine learning techniques on features extracted from the malware's code and behavior in order to identify other similar malware whose authors are known. However, the same malware can be reused by multiple actors, and the actor who performed an attack using a malware might differ from the malware's author. Moreover, information collected during an incident may contain many clues about the identity of the attacker in addition to the malware used. In this paper, we propose a method of attack attribution based on textual analysis of threat intelligence reports, using state of the art algorithms and models from the fields of machine learning and natural language processing (NLP). We have developed a new text representation algorithm which captures the context of the words and requires minimal feature engineering. Our approach relies on vector space representation of incident reports derived from a small collection of labeled reports and a large corpus of general security literature. Both datasets have been made available to the research community. Experimental results show that the proposed representation can attribute attacks more accurately than the baselines' representations. In addition, we show how the proposed approach can be used to identify novel previously unseen threat actors and identify similarities between known threat actors.

When evaluating algorithms that recommend a list of relevant items to a user, it is common to use metrics such as precision to measure the system accuracy. When computing precision, one computes the number of items that were selected by the user among the recommended items. As such, recommended items that were not selected by the user, which we call \em rejected recommendations, are all considered to be bad recommendations, resulting in no increase to the system accuracy metric. Our ultimate goal is to develop a new recommendation accuracy evaluation metric, which may assign some value to the rejected recommendations. In this paper, as a first step, we claim that some rejected recommendations are better than others. Specifically, we consider items that are similar to the item that was finally selected, as better recommendations than items that bear little similarity. We conduct a user study, showing that rejected recommendations that have high content or collaborative similarity to the selected item are perceived by users as better recommendations than items with low similarity. In addition, we study the correlations between the recommended items shown to a user and the un-recommended items that the user has selected in a real-life job posting dataset. We show that when considering item similarity rather than simple precision, the correlations are much higher. This may be attributed to the influence of the recommended items on the decisions of the user.

One of the challenges of automating machine learning applications is the automatic selection of an algorithmic model for a given problem. We present AutoDi, a novel and resource-efficient approach for model selection. Our approach combines two sources of information: metafeatures extracted from the data itself and word-embedding features extracted from a large corpus of academic publications. This hybrid approach enables AutoDi to select top-performing algorithms both for widely and rarely used datasets by utilizing its two types of feature sets. We demonstrate the effectiveness of our proposed approach on a large dataset of 119 datasets and 179 classification algorithms grouped into 17 families. We show that AutoDi can reach an average of 98.8% of optimal accuracy and select the optimal classification algorithm in 49.5% of all cases.

Deep learning is now an integral part of recommender systems, but the research is still in its early phase. New research topics pop up frequently and established topics are extended in new, interesting directions. DLRS 2018 is a venue for pioneering work in the intersection of deep learning and recommender systems research.

The phenomenon of trolling has emerged as a widespread form of abuse on news sites, online social networks, and other types of social media. In this paper, we study a particular type of trolling, performed by asking a provocative question on a community question-answering website. By combining user reports with subsequent moderator deletions, we identify a set of over 400,000 troll questions on Yahoo Answers, i.e., questions aimed to inflame, upset, and draw attention from others on the community. This set of troll questions spans a lengthy period of time and a diverse set of topical categories. Our analysis reveals unique characteristics of troll questions when compared to "regular" questions, with regards to their metadata, text, and askers. A classifier built upon these features reaches an accuracy of 85% over a balanced dataset. The answers' text and metadata, reflecting the community's response to the question, are found particularly productive for the classification task.

Inferring socio-demographic attributes of users is an important and challenging task that could help with personalization, recommendation, advertising, etc. Sensor data collected from mobile devices can be utilized for inferring such attributes. Previous works have focused on combining different types of sensors, such as applications, accelerometer, GPS, battery, and many others, to achieve this task. In this study, we were able to infer attributes, such as gender, age, marital status, and whether the user has children, using solely the GPS sensor. We suggest a novel inference technique, which learns an embedding representation of preprocessed spatial GPS trajectories using an adaption of the Word2vec approach. Based on the embedding representation, we later train multiple classification models to achieve the inference goals. Our empirical results indicate that the suggested embedding approach outperforms a classification approach which does not take into consideration the embedding patterns. Experiments on real datasets collected from Android devices show that the proposed method achieves over 80% accuracy for various demographic prediction tasks.

Questions on community question answering websites usually reflect one of two intents: learning information or starting a conversation. In this paper, we revisit this fundamental classification task of informational versus conversational questions, which was originally introduced and studied in 2009. We use a substantially larger dataset of archived questions from Yahoo Answers, which includes the question»s title, description, answers, and votes. We replicate the original experiments over this dataset, point out the common and different from the original results, and present a broad set of characteristics that distinguish the two question types. We also develop new classifiers that make use of additional data types, advanced machine learning, and a large dataset of unlabeled data, which achieve enhanced performance.

Deep learning methods became widely popular in the recommender systems community in 2016, in part thanks to the previous event of the DLRS workshop series. Now, deep learning has been embedded in the main conference as well and initial research directions have started forming, so the role of DLRS 2017 is to encourage starting new research directions, incentivize the application of very recent techniques from deep learning, and provide a venue for specialized discussion of this topic.

Smartphones have become ubiquitous in our daily lives; they are used for a wide range of tasks and store increasing amounts of personal data. To minimize risk and prevent misuse of this data by unauthorized users, access must be restricted to verified users. Current classification-based methods for gesture-based user verification only consider single gestures, and not sequences. In this paper, we present a method which utilizes information from sequences of touchscreen gestures, and the context in which the gestures were made. To evaluate our approach, we built an application which records all the necessary data from the device (touch and contextual sensors which do not consume significant battery life), and installed it on several Galaxy S4 smartphones. The smartphones were given to 20 volunteers to use as their personal phones for two-weeks. Using XGBoost on the collected data, we were able to classify between a legitimate user and the population of illegitimate users (imposters) with an average equal error rate (EER) of 4.78% and an average area under the curve (AUC) of 98.15%. Our method demonstrates that by considering sequences of gestures, as opposed to individual gestures, the accuracy of the verification process improves significantly.

Context-aware systems enable the sensing and analysis of user context in order to provide personalized services. Our study is part of growing research efforts examining how high-dimensional data collected from mobile devices can be utilized to infer users' dynamic preferences. We present a novel method for inferring contextual user preferences by using an unsupervised deep learning technique applied to mobile sensor data. We train an auto-encoder for each user preference with contextual data that based on past user interaction with the system. Given new contextual sensor data from a user, the patterns discovered from each auto-encoder are used to predict the most likely preference in the given context. This can greatly enhance a variety of services, such as mobile online advertising and context-aware recommender systems. We demonstrate our contribution with a point of interest (POI) recommender system in which we label contextual preferences based on the interaction of users with categories of items. Empirical results utilizing a real world dataset of mobile users show a significant improvement (16% to 73% improvement) in classification accuracy compared with state of the art classification methods.

Attack propagation models within honeypot systems aim at providing insights about attack strategies that target multiple honeypots, rather than analyzing attacks on each honeypot separately. Traditional attack propagation models focus on building a single probabilistic model. This modeling approach may be misleading, since it does not take into consideration contextual information such as the country from which the attack is initiated. In addition, with the massive increase in the magnitude of attacks on honeypots, a scalable modeling approach is required. In this work we present a novel attack propagation model that can utilize contextual information about the attacks by training multiple Markov Chain models. Moreover, we add additional layers of analysis: first, we present a likelihood estimation procedure that can identify new and evolving attack patterns; and second, we introduce a method for generating simulated attack sequences that can be used for training or sensitivity analysis. Lastly, we present, in details, a MapReduce design for all suggested algorithms in order to address scalability issues. We evaluate our methods on a massive dataset which includes approximately 170 million attacks on an operational honeypot system. Results indicate that contextual modeling is important for explaining attack propagation that may vary by country. In addition, we show the effectiveness of the suggested method for generating simulated sequences by comparing the attack propagation patterns we learned in the generated dataset and the original one. Finally, we demonstrate the scalability of all of the proposed algorithms on real and synthetic datasets that include over a billion records.

Security systems for databases produce numerous alerts about anomalous activities and policy rule violations. Prioritizing these alerts will help security personnel focus their efforts on the most urgent alerts. Currently, this is done manually by security experts that rank the alerts or define static risk scoring rules. Existing solutions are expensive, consume valuable expert time, and do not dynamically adapt to changes in policy. Adopting a learning approach for ranking alerts is complex due to the efforts required by security experts to initially train such a model. The more features used, the more accurate the model is likely to be, but this will require the collection of a greater amount of user feedback and prolong the calibration process. In this paper, we propose CyberRank, a novel algorithm for automatic preference elicitation that is effective for situations with limited experts' time and outperforms other algorithms for initial training of the system. We generate synthetic examples and annotate them using a model produced by Analytic Hierarchical Processing (AHP) to bootstrap a preference learning algorithm. We evaluate different approaches with a new dataset of expert ranked pairs of database transactions, in terms of their risk to the organization. We evaluated using manual risk assessments of transaction pairs, CyberRank outperforms all other methods for cold start scenario with error reduction of 20%.

We believe that Deep Learning is one of the next big things in Recommendation Systems technology. The past few years have seen the tremendous success of deep neural networks in a number of complex tasks such as computer vision, natural language processing and speech recognition. Despite this, only little work has been published on Deep Learning methods for Recommender Systems. Notable recent application areas are music recommendation, news recommendation, and session-based recommendation. The aim of the workshop is to encourage the application of Deep Learning techniques in Recommender Systems, to promote research in deep learning methods for Recommender Systems, and to bring together researchers from the Recommender Systems and Deep Learning communities.

Recommender systems (RS) can now be found in many commercial Web sites, often presenting customers with a short list of additional products that they might purchase. Many commercial sites do not typically have the ability and resources to develop their own system and may outsource the RS to a third party. This had led to the growth of a recommendation as a service industry, where companies, referred to as RS providers, provide recommendation services. These companies must carefully balance the cost of building recommendation models and the payment received from the e-business, as these payments are expected to be low. In such a setting, restricting the computational time required for model building is critical for the RS provider to be profitable. In this article, we propose anytime algorithms as an attractive method for balancing computational time and the recommendation model performance, thus tackling the RS provider problem. In an anytime setting, an algorithm can be stopped after any amount of computational time, always ensuring that a valid, although suboptimal, solution will be returned. Given sufficient time, however, the algorithm should converge to an optimal solution. In this setting, it is important to evaluate the quality of the returned solution over time, monitoring quality improvement. This is significantly different from traditional evaluation methods, which mostly estimate the performance of the algorithm only after its convergence is given sufficient time. We show that the popular item-item top-N recommendation approach can be brought into the anytime framework by smartly considering the order by which item pairs are being evaluated. We experimentally show that the time-accuracy trade-off can be significantly improved for this specific problem.

Online social networks are a popular and important channel for people to share, find and disseminate information on a massive scale. Some of the information exposed through these networks is meant to be private. However, sensitive organizational information can be accidentally leaked by employees and become exposed to adversaries or competitors. The threat is escalated due to socialbots used by adversaries to penetrate the informal social network of an organization's employees in order to harvest sensitive information. This study evaluates the ability of an attacker to harvest leaked information using socialbots versus the effort required to wire the profiles into the organizational network. The evaluation is performed using real information diffusion data of two social networks and extensive simulations of socialbot wiring strategies. Our results demonstrate that organizations whose social network topologies are characterized by low clustering coefficient are more vulnerable to eavesdropping. We also show that the most effective socialbot wiring strategy for harvesting information is different from the most effective strategies for infiltrating the organization.

Wikipedia - the online encyclopedia - has long been used as a source of information for researchers, as well as being a subject of research itself. Wikipedia has been shown to be effective in recommender systems, sentiment analysis, validation and multiple domains in information retrieval. One of the reasons for Wikipedia's popularity among researchers and practitioners is the multiple types of information it contains, which enables practitioners to select the right "tool" for their respective tasks. In addition to its great potential, this multitude of information sources also poses a challenge: which sources of information are best suited for a specific problem and how can different types of data be combined? This tutorial aims to provide a holistic view of Wikipedia's different features - text, links, categories, page views, editing history etc. - and explore the different ways they can be utilized in a machine learning framework. By presenting and contrasting the latest works that utilize Wikipedia in multiple domains, this tutorial aims to increase the awareness among researchers and practitioners in these fields to the benefits of utilizing Wikipedia in their respective domains, in particular to the use of multiple sources of information simultaneously.

Most work in heuristic search focused on path finding problems in which the cost of a path in the state space is the sum of its edges' weights. This paper addresses a different class of path finding problems in which the cost of a path is the product of its weights. We present reductions from different classes of multiplicative path finding problems to suitable classes of additive path finding problems. As a case study, we consider the problem of finding least and most probable paths in a Markov Chain, where path cost corresponds to the probability of traversing it. The importance of this problem is demonstrated in an anomaly detection application for cyberspace security. Three novel anomaly detection metrics for Markov Chains are presented, where computing these metrics require finding least and most probable paths. The underlying Markov Chain is dynamically changing, and so fast methods for computing least and most probable paths are needed. We propose such methods based on the proposed reductions and using heuristic search algorithms.

A group may appreciate recommendations on items that fit their joint preferences. When the members' actual preferences are unknown, a recommendation can be made with the aid of collaborative filtering methods. We offer to narrow down the recommended list of items by eliciting the users' actual preferences. Our final goal is to output top-k preferred items to the group out of the top-N recommendations provided by the recommender system (k

The query-performance prediction task is to estimate retrieval effectiveness with no relevance judgments. Pre-retrieval prediction methods operate prior to retrieval time. Hence, these predictors are often based on analyzing the query and the corpus upon which retrieval is performed. We propose a {\em corpus-independent} approach to pre-retrieval prediction which relies on information extracted from Wikipedia. Specifically, we present Wikipedia-based features that can attest to the effectiveness of retrieval performed in response to a query {\em regardless} of the corpus upon which search is performed. Empirical evaluation demonstrates the merits of our approach. As a case in point, integrating the Wikipedia-based features with state-of-the-art pre-retrieval predictors that analyze the corpus yields prediction quality that is consistently better than that of using the latter alone.

In this paper we present Nesto, a network selection and offloading system for android based mobile devices. Nesto chooses the best connectivity solution between available heterogeneous wireless networks using network switching. The suggested framework supports several configurable policies and addresses the following requirements: battery energy saving, bandwidth maximization, an offloading strategy for cellular operators and granting the best available network QoS to current running applications (e.g. minimizing delay and jitter for voip applications). Nesto is designed to support two primary connectivity modes: a traditional single connectivity mode and a full dual mode, where both the cellular and ad-hoc WiFi networks are used simultaneously. The full dual mode allows us to extend the always best connected definition from the device level to the application level, i.e.: selecting the best network for each application. This paper presents the architecture of Nesto and the different network selection optimization models. We evaluate our solution with simulated data and with real network traffic traces. Preliminary results indicate that: (1) energy efficient policies rely on the single connectivity operation mode, but they can be controlled to improve other networking QoS measures with minimum energy overhead, (2) using the full dual operation mode improves the overall networking performances of the device, (3) the full dual operation mode enables an efficient always best connected solution at the application level, optimizing the relevant measures for each application type.

In this paper, we present a method for boosting collaborative filtering by integrating spatial information about geo-referenced items (e.g., photos). In particular, we developed a method to estimate missing ratings by propagating an item's neighbor's ratings based on the similarity of geospatial information. An empirical evaluation shows that geospatial information significantly improves recommendation results, and its contribution grows with the ratings data's level of sparseness. We illustrate the usefulness of the method for a photo recommendation task using data obtained from two popular photo-sharing websites: Flickr and Panoramio. A comparison with state-of-the-art methods indicates the superiority of the proposed method, implying that geospatial information should be considered, when available.

Providing adaptive help during interaction with the system can be used to assist users in accomplishing their tasks. We propose providing guidance by highlighting the steps required for performing a task that the user intends to complete according to the prediction of a system. We present a study aimed at examining whether highlighting intended user steps in menus and toolbars as a means of assisting users in performing tasks is useful in terms of user response and performance. We also examined the effects of different accuracy levels of help and the control format on user response and performance. An experiment was conducted in which 64 participants performed tasks using menus and toolbars of a simulated email application. Participants were offered a highlighted guidance of the required steps in varying levels of accuracy (100%, 80%, 60% and no guidance). Our results support the benefits of highlighted help both in user performance times and in user satisfaction from receiving such assistance. Users found the assistance necessary and helpful and by the same token not unduly intrusive. Additionally, users felt that such assistance generally helped in reducing performance time on tasks. We did not find a significant difference when users receiving help at 80% accuracy was compared to those receiving help at 100% accuracy; however, such a difference does appear for those receiving 60% accuracy. In such cases, we found that the user's satisfaction level, perceived usefulness and trust in the system decreased while their notion of perceived intrusiveness increased. We conclude that assisting users by highlighting the required steps is useful so long as the minimal accuracy level of help is higher than 60%. Our study has implications on the implementation of highlighting next steps as a means of adaptive help and on integrating probability-based algorithms such as intention prediction to adaptive assistance systems.

Recommendation systems often compute fixed‐length lists of recommended items to users. Forcing the system to predict a fixed‐length list for each user may result in different confidence levels for the computed recommendations. Reporting the system's confidence in its predictions (the recommendation strength) can provide valuable information to users in making their decisions. In this article, we investigate several different displays of a system's confidence to users and conclude that some displays are easier to understand and are favored by most users. We continue to investigate the effect confidence has on users in terms of their perception of the recommendation quality and the user experience with the system. Our studies show that it is not easier for users to identify relevant items when confidence is displayed. Still, users appreciate the displays and trust them when the relevance of items is difficult to establish.

Most collaborative Recommender Systems (RS) operate in a single domain (such as movies, books, etc.) and are capable of providing recommendations based on historical usage data which is collected in the specific domain only. Cross-domain recommenders address the sparsity problem by using Machine Learning (ML) techniques to transfer knowledge from a dense domain into a sparse target domain. In this paper we propose a transfer learning technique that extracts knowledge from multiple domains containing rich data (e.g., movies and music) and generates recommendations for a sparse target domain (e.g., games). Our method learns the relatedness between the different source domains and the target domain, without requiring overlapping users between domains. The model integrates the appropriate amount of knowledge from each domain in order to enrich the target domain data. Experiments with several datasets reveal that, using multiple sources and the relatedness between domains improves accuracy of results.

In this paper, we introduce a novel approach to generate an intention prediction model of user interactions with systems. As part of this new approach, we include personal aspects, such as user characteristics, that can increase prediction accuracy. The model is automatically trained according to the user's fixed attributes (e.g., demographic data such as age and gender) and the user's sequences of actions in the system. The generated model has a tree structure. The building blocks of each node can be any probabilistic sequence model [such as hidden Markov models (HMMs) and conditional random fields (CRFs)] and each node is split according to user attributes. Thus, we refer to this algorithm as an attribute-driven model tree. The new model was first tested on simulated data in which users with different attributes (such as age and gender) behave differently when trying to accomplish various tasks. We then validated the ability of the algorithm to discover the relevant attributes. We tested our algorithm on two real datasets: from a web application and a mobile application dataset. The results were encouraging and indicate the capability of the proposed method to discover the correct user intention model and increasing intention prediction accuracy compared with single HMM or CRF models.

In this demo we present a recommender benchmark framework that serves as an infrastructure for comparing and examining the performance and feasibility of different recommender algorithms on various datasets with a variety of measures. The extendable infrastructure aims to provide easy plugging of novel recommendation-algorithms, datasets and compare their performance using visual tools and metrics with other algorithms in the benchmark. It also aims at generating a WEKA-type workbench [1] for the recommender systems field to enable usage and application of common recommender systems (RS) algorithms for research and practice. The demo movie is available at: http://www.youtube.com/watch?v=fsDITf6s0WY

One important challenge in the field of recommender systems is the sparsity of available data. This problem limits the ability of recommender systems to provide accurate predictions of user ratings. We overcome this problem by using the publicly available user generated information contained in Wikipedia. We identify similarities between items by mapping them to Wikipedia pages and finding similarities in the text and commonalities in the links and categories of each page. These similarities can be used in the recommendation process and improve ranking predictions. We find that this method is most effective in cases where ratings are extremely sparse or nonexistent. Preliminary experimental results on the MovieLens dataset are encouraging.

Despite improvements in their capabilities, search engines still fail to provide users with only relevant results. One reason is that most search engines implement a “one size fits all” approach that ignores personal preferences when retrieving the results of a user's query. Recent studies (Smyth, 2010) have elaborated the importance of personalizing search results and have proposed integrating recommender system methods for enhancing results using contextual and extrinsic information that might indicate the user's actual needs. In this article, we review recommender system methods used for personalizing and improving search results and examine the effect of two such methods that are merged for this purpose. One method is based on collaborative users' knowledge; the second integrates information from the user's social network. We propose new methods for collaborative-and social-based search and demonstrate that each of these methods, when separately applied, produce more accurate search results than does a purely keyword-based search engine (referred to as “standard search engine”), where the social search engine is more accurate than is the collaborative one. However, separately applied, these methods do not produce a sufficient number of results (low coverage). Nevertheless, merging these methods with those implemented by standard search engines overcomes the low-coverage problem and produces personalized results for users that display significantly more accurate results while also providing sufficient coverage than do standard search engines. The improvement, however, is significant only for topics for which the diversity of terms used for queries among users is low.

Group Recommendation Systems (GRS) aim at recommending items that are relevant for the joint interest of a group of users. Voting mechanisms assume that users rate all items in order to identify an item that suits the preferences of all group members. This assumption is not feasible in sparse rating scenarios which are common in the recommender systems domain. In this paper we examine an application of voting theory to GRS. We propose a method to accurately determine the winning item while using a minimal set of the group members ratings, assuming that the recommender system has probabilistic knowledge about the distribution of users' ratings of items in the system. Since computing the optimal minimal set of ratings is computationally intractable, we propose two heuristic algorithms that proceed iteratively that aiming atto minimizing the number of required ratings, until identifying a "winning item". Experiments with the Netflix data show that the proposed algorithms reduce the required number of ratings for identifying the "winning item" by more than 50%.

Abstract Terrorist groups use the Web as their infrastructure for various purposes. One example is the forming of new local cells that may later become active and perform acts of terror. The Advanced Terrorist Detection System (ATDS), is aimed at tracking down online access to abnormal content, which may include terrorist‐generated sites, by analyzing the content of information accessed by the Web users. ATDS operates in two modes: the training mode and the detection mode. In the training mode, ATDS determines the typical interests of a prespecified group of users by processing the Web pages accessed by these users over time. In the detection mode, ATDS performs real‐time monitoring of the Web traffic generated by the monitored group, analyzes the content of the accessed Web pages, and issues an alarm if the accessed information is not within the typical interests of that group and similar to the terrorist interests. An experimental version of ATDS was implemented and evaluated in a local network environment. The results suggest that when optimally tuned the system can reach high detection rates of up to 100% in case of continuous access to a series of terrorist Web pages.

Many applications that employ data mining techniques involve mining data that include private and sensitive information about the subjects. One way to enable effective data mining while preserving privacy is to anonymize the data set that includes private information about subjects before being released for data mining. One way to anonymize data set is to manipulate its content so that the records adhere to k-anonymity. Two common manipulation techniques used to achieve k-anonymity of a data set are generalization and suppression. Generalization refers to replacing a value with a less specific but semantically consistent value, while suppression refers to not releasing a value at all. Generalization is more commonly applied in this domain since suppression may dramatically reduce the quality of the data mining results if not properly used. However, generalization presents a major drawback as it requires a manually generated domain hierarchy taxonomy for every quasi-identifier in the data set on which k-anonymity has to be performed. In this paper, we propose a new method for achieving k-anonymity named K-anonymity of Classification Trees Using Suppression (kACTUS). In kACTUS, efficient multidimensional suppression is performed, i.e., values are suppressed only on certain records depending on other attribute values, without the need for manually produced domain hierarchy trees. Thus, in kACTUS, we identify attributes that have less influence on the classification of the data records and suppress them if needed in order to comply with k-anonymity. The kACTUS method was evaluated on 10 separate data sets to evaluate its accuracy as compared to other k-anonymity generalization- and suppression-based methods. Encouraging results suggest that kACTUS' predictive performance is better than that of existing k-anonymity algorithms. Specifically, on average, the accuracies of TDS, TDR, and kADET are lower than kACTUS in 3.5, 3.3, and 1.9 percent, respectively, despite their usage of manually defined domain trees. The accuracy gap is increased to 5.3, 4.3, and 3.1 percent, respectively, when no domain trees are used.

Abstract This paper describes ePaper, a research prototype system of a personalized newspaper on a mobile reading device. The ePaper aggregates content (i.e., news items) from various news providers, classifies the news items according to concepts from a news domain ontology, and delivers an electronic newspaper to each subscribed user (reader). The system personalizes the content of the newspaper according to the user's profiles and preferences by applying ontological content‐based and collaborative filtering algorithms. The user's profile is updated implicitly and dynamically, based on the tracking of their reading. Beyond personalization, the ePaper can also provide the user with a “standard edition” of a selected newspaper, as well as browsing capabilities in a repository of news items. The layout of the newspaper is adapted to the specifications of the reading device and to the user's preferences. In this overview paper, we highlight the main research challenges involved in the development of ePaper and describe how we addressed them.

A new ontological-content-based method for ranking the relevancy of items in the electronic newspapers domain is proposed. The method is being implemented in ePaper, a personalized electronic newspaper research project. The content-based part of the filtering method of ePaper utilizes a hierarchical ontology of news items. The method considers common and "close" ontology concepts appearing in the user's profile and in the item's profile, measuring the hierarchical distance between concepts in the two profiles. Based on the number of common and related concepts, and their distances from each other, the filtering algorithm computes the similarity between items and users, and rank-orders the news items according to their relevancy to each user, thus providing a personalized newspaper.

Abstract The outputs of several information filtering (IF) systems can be combined to improve filtering performance. In this article the authors propose and explore a framework based on the so‐called information structure (IS) model, which is frequently used in Information Economics, for combining the output of multiple IF systems according to each user's preferences (profile). The combination seeks to maximize the expected payoff to that user. The authors show analytically that the proposed framework increases users expected payoff from the combined filtering output foranyuser preferences. An experiment using the TREC‐6 test collection confirms the theoretical findings.

Abstract Web navigation enables easy access to vast amounts of information and services. However, it also poses a major risk to users' privacy. Various eavesdroppers constantly attempt to violate users' privacy by tracking their navigation activities and inferring their interests and needs (profiles). Users who wish to keep their intentions secret forego useful services to avoid exposure. The computer security community has concentrated on improving users' privacy by concealing their identity on the Web. However, users may want or need to identify themselves over the Net to receive certain services but still retain their interests, needs, and intentions in private. PRAW—a PRivAcy model for the Web suggested in this paper—is aimed at hiding users' navigation tracks to prevent eavesdroppers from inferring their profiles but still allowing them to be identified. PRAW is based on continuous generation of fake transactions in various fields of interests to confuse eavesdroppers' automated programs, thus providing them false data. A privacy measure is defined that reflects the difference between users' actual profile and the profile that eavesdroppers might infer. A prototype system was developed to examine PRAW's feasibility and conduct experiments to test its effectiveness. Encouraging results and their analysis are presented, as well as possible attacks and known limitations.

The paper describes a user study examining methods for improving users queries, specifically interactive and automatic query expansion and advanced search options. The user study includes subjective and objective evaluation of the effect of the above methods and a comparison between the real and perceived effect.

Abstract In collaborative information finding systems, evaluations provided by users assist other users with similar needs. This article examines the problem of getting users to provide evaluations, thus overcoming the so‐called “free‐riding” behavior of users. Free riders are those who use the information provided by others without contributing evaluations of their own. This article reports on an experiment conducted using the “AntWorld,” system, a collaborative information finding system for the Internet, to explore the effect of added motivation on users' behavior. The findings suggest that for the system to be effective, users must be motivated either by the environment, or by incentives within the system. The findings suggest that relatively inexpensive extrinsic motivators can produce modest but significant increases in cooperative behavior.