Peer review is central to scientific decision-making, yet it is rarely evaluated or audited at scale. Growing submission volumes and the increasing use of large language models (LLMs) in drafting reviews have introduced new challenges for transparency, accountability, and quality control. Today, peer reviews are often produced through hybrid human--AI workflows, where a reviewer may develop the core evaluative ideas while using an LLM to refine wording, restructure arguments, or improve fluency. This shift raises new questions beyond authorship detection alone: Are reviews constructive? Are reviewer claims grounded in the submitted paper? How can we quantify collaboration between human reasoning and AI-assisted writing, and distinguish whether the intellectual contribution or the surface text originates from humans or models?

Large Language Models (LLMs) are now widely used for query reformulation and expansion in Information Retrieval, with many studies reporting substantial effectiveness gains. However, these results are typically obtained under heterogeneous experimental conditions, making it difficult to assess which findings are reproducible and which depend on specific implementation choices. In this work, we present a systematic reproducibility and comparative study of ten representative LLM-based query reformulation methods under a unified and strictly controlled experimental framework. We evaluate methods across two architectural LLM families at two parameter scales, three retrieval paradigms (lexical, learned sparse, and dense), and nine benchmark datasets spanning TREC Deep Learning and BEIR. Our results show that reformulation gains are strongly conditioned on the retrieval paradigm, that improvements observed under lexical retrieval do not consistently transfer to neural retrievers, and that larger LLMs do not uniformly yield better downstream performance. These findings clarify the stability and limits of reported gains in prior work. To enable transparent replication and ongoing comparison, we release all prompts, configurations, evaluation scripts, and run files through QueryGym, an open-source reformulation toolkit with a public leaderboard.\footnote{https://leaderboard.querygym.com}

Peer review is central to scientific publishing, yet reviewers frequently include claims that are subjective, rhetorical, or misaligned with the submitted work. Assessing whether review statements are factual and verifiable is crucial for fairness and accountability. At the scale of modern conferences and journals, manually inspecting the grounding of such claims is infeasible. We present Peerispect, an interactive system that operationalizes claim-level verification in peer reviews by extracting check-worthy claims from peer reviews, retrieving relevant evidence from the manuscript, and verifying the claims through natural language inference. Results are presented through a visual interface that highlights evidence directly in the paper, enabling rapid inspection and interpretation. Peerispect is designed as a modular Information Retrieval (IR) pipeline, supporting alternative retrievers, rerankers, and verifiers, and is intended for use by reviewers, authors, and program committees. We demonstrate Peerispect through a live, publicly available demo at https://app.reviewer.ly/app/peerispect and API services at https://github.com/Reviewerly-Inc/Peerispect, accompanied by a video tutorial https://www.youtube.com/watch?v=pc9RkvkUh14.

Modern information retrieval pipelines combine retrieval, LLM-based generation, and LLM-based judgment, and a poor outcome may originate in any of the three stages. Existing work studies these failures in isolation. This paper instead asks whether query difficulty itself transfers across the three stages: are the same queries hard to retrieve, hard to generate for, and hard to judge? Using four years of TREC Deep Learning benchmarks (2019–2022), we define hard-to-retrieve, hard-to-generate, and hard-to-judge query sets under a unified quartile-based operationalization and analyze their overlap, their stability across system configurations, and the linguistic and semantic causes of failure in each task. We find that the three sets overlap only weakly; three-way overlap is at or below the level expected under independence, indicating that difficulty is largely task-conditioned and does not transfer reliably across stages. The overlap structure is nonetheless stable across retrievers, generators, and judging setups, suggesting that task-specific difficulty is driven by query characteristics interacting with each task's inductive biases rather than by model choice. We further induce a data-driven typology of failure causes and show that conditioning generation on task-relevant difficulty cues yields consistent gains in answer quality.

Large Language Models (LLMs) are increasingly used in scientific peer review, assisting with drafting, rewriting, and refinement. However, existing peer-review LLM detection methods largely treat authorship as a binary problem (human vs. AI) without accounting for the hybrid nature of modern review workflows. In practice, evaluative ideas and surface realization may originate from different sources, creating a spectrum of human-AI collaboration. To address this, we introduce PeerPrism, a large-scale benchmark of 20,690 peer reviews explicitly designed to disentangle idea provenance from text provenance. We construct controlled generation regimes spanning fully human, fully synthetic, and multiple hybrid transformations. We benchmark state-of-the-art LLM text detection methods on PeerPrism. While several methods achieve high accuracy on the standard binary task, their predictions diverge sharply under hybrid regimes. In particular, when ideas originate from humans but the surface text is AI-generated, detectors frequently disagree and produce contradictory classifications. Our results show that current detection methods conflate surface realization with intellectual contribution. Rather than relying on this binary, authorship must be modeled as a multidimensional construct spanning semantic reasoning and stylistic realization. PeerPrism is the first benchmark evaluating human-AI collaboration in these settings. We release all code, data, prompts, and evaluation scripts to facilitate reproducible research at https://github.com/Reviewerly-Inc/PeerPrism.

While large language models (LLMs) have been widely studied in scholarly workflows, e.g., for citation recommendation and literature summarization and more, their role in supporting research integrity remains underexplored. In this talk, we share our experience building and deploying two real-world systems that audit peer reviews and verify their factual grounding at scale. We evaluate the capabilities and limitations of LLMs in two key tasks: (1) assessing review quality along dimensions like specificity and tone, and (2) verifying whether reviewer claims are supported by the submitted paper. Using expert-annotated benchmarks, we compare static metrics, ML baselines, zero-shot LLMs, and fine-tuned models to assess alignment with human judgment. The talk will highlight methodological choices, deployment lessons, and empirical insights into where LLMs succeed and where hybrid approaches with interpretable ML and retrieval perform more reliably. We conclude with reflections on what infrastructure is needed to make use LLMs as a robust foundation for research integrity at scale.

Neural rankers often reflect societal biases (e.g., gender) present in training data. Existing methods typically require protected-attribute labels or model modifications. We introduce a simple, model-agnostic approach that uses the model's loss as a bias proxy within a self-paced learning curriculum. By prioritizing lower-loss examples and gradually incorporating harder ones, our method reduces reliance on biased data without demographic annotations. We theoretically show a monotonic decrease in the gender loss gap and empirically demonstrate reduced bias with comparable or improved ranking performance on MS MARCO, outperforming strong baselines.

Background Artificial intelligence (AI) is increasingly embedded in social and institutional decision-making, creating demand for ethically literate practitioners. Universities have responded by introducing AI ethics instruction, but the structure, content, pedagogy, and evaluation of these efforts remain unevenly documented. Objective To map and synthesize research on university level AI ethics education by characterizing course design, pedagogy, ethical themes, and assessment methods, and identifying evidence gaps that limit knowledge consolidation and instructional refinement. Methods We conducted a scoping review using Continuous Active Learning to screen 50,766 records up to 2024. 43 studies met inclusion after title, abstract, and full text review. We coded instructional design, curricular themes, pedagogical methods, and evaluation approaches using descriptive frequency counts and qualitative synthesis. Results Most included studies were conceptual or descriptive, with relatively few empirical evaluations. Instruction was concentrated in computing and engineering and primarily targeted undergraduate learners. Ethics content was more often embedded within technical courses than delivered as standalone offerings. Reported pedagogy relied heavily on lecture and case-based discussion, with fewer studies describing participatory formats such as simulations or role-play. Curricular emphasis clustered around bias/fairness and privacy, with comparatively less attention to governance, explainability, and trust. Evaluation most often relied on self-report and reflective methods, while validated instruments and performance-based assessments were less common, and behavioral or applied outcomes were rarely assessed. Conclusions The literature suggests a field oriented toward awareness-building more than measurable ethical competence. Clearer competency claims, stronger assessment transparency, and greater alignment between instructional design and evaluation would improve comparability across studies and support evidence-informed course development.

Forming effective expert teams is central to domains where solving complex problems requires diverse, complementary skills. However, automating this task is highly challenging due to sparse co-occurrence data, long-tailed expert participation, and the combinatorial complexity of unseen skill configurations. Existing graph-based, probabilistic, and neural approaches often struggle with generalization, fairness, and robustness, leading to biased selections that favor historically popular experts over more suitable candidates. To address these challenges, we propose a generative framework for expert team formation based on denoising diffusion probabilistic models. We cast team formation as skill-conditioned imputation (i.e., inpainting), where skills are treated as observed context and the expert component is generated via conditional diffusion sampling. This design enables our method to preserve semantic skill–expert alignment, mitigate data sparsity, and generate diverse yet contextually coherent teams. Extensive experiments on DBLP and DOTA2 datasets show that our model consistently outperforms state-of-the-art baselines, achieving over 3 × higher recall (16.4% vs. 5.0%) and MAP (9.7% vs. 2.2%) on DBLP, while delivering more than 5 × improvement in MRR (13.3% vs. 2.5%) on DOTA2. Fairness analysis further demonstrates that our method reduces average overlap with the top-100 most popular experts to 2.6, compared to 86.7 for the strongest baseline, and achieves near-optimal diversity with NDKL ≈ 0.1 under high non-popular expert ratios. For reproducibility purposes , we made our code and model publicly available at https://github.com/17shiraz/DiffTF .

The varying performance of information retrieval (IR) methods, including state-of-the-art transformer-based neural retrievers, across diverse queries poses a significant challenge for achieving robust and reliable retrieval effectiveness. Query Performance Prediction (QPP) seeks to estimate the effectiveness of a retrieval method for individual queries, enabling adaptive strategies to improve retrieval outcomes, particularly for challenging queries. However, existing QPP approaches face fundamental challenges: pre-retrieval methods often rely on surface-level query features that fail to capture the nuanced relationship between queries and retrieval effectiveness, while post-retrieval methods depend heavily on the quality of retrieved documents, which can be unreliable for difficult queries. To this end, we propose the Query Space Distance-Based QPP ( QSD-QPP ) framework, which leverages the deterministic and consistent behavior of retrieval methods to estimate query performance by referencing historical queries with known effectiveness. The approach is motivated by the observation that semantically or syntactically similar queries often exhibit consistent retrieval performance, a property that can be exploited to make reliable predictions for unseen queries. QSD-QPP operates in two modes: (1) a lightweight pre-retrieval instantiation that dynamically constructs a query subspace based on embedding distances to interpolate the performance of proximate historical queries, and (2) an enriched post-retrieval instantiation that incorporates contextualized embeddings, document interactions, and historical query associations to enhance prediction accuracy. By utilizing large-scale contextualized embeddings derived from pre-trained language models, QSD-QPP efficiently identifies semantically similar queries and leverages their performance for robust predictions. By addressing the inherent limitations of prior approaches, QSD-QPP achieves a balanced trade-off between computational efficiency, prediction accuracy, and scalability. We evaluate QSD-QPP on four benchmark datasets, including MS MARCO Dev and TREC Deep Learning tracks (2019, 2020, and DL-Hard), demonstrating its superior accuracy and robustness compared to state-of-the-art baselines in both pre-retrieval and post-retrieval QPP tasks. To ensure reproducibility and encourage further research, we publicly release the implementation of our work.

The task of searching over large keyword graphs aims to identify a subgraph where the nodes collectively cover the input query keywords. Although finding an exact solution to this problem is NP-hard, we address it by proposing a novel graph neural network representation learning technique specifically tailored for graphs with missing information. We propose a novel keyword graph representation learning method that incorporates complementary aspects of graphs: global, local, adjusted, and feature semantics. Considering these multiple aspects, our approach remains robust and resilient to missing information. We adopt and fine-tune a transformer-based model to aggregate the various features of a graph to generate rich representations, recognizing the pivotal role of keywords in this task. We show through experiments on real-world data that our method outperforms the state-of-the-art approaches and is particularly robust in the face of missing values, underscoring its ability to effectively handle incomplete graphs.

Ad hoc retrieval, a cornerstone task in Information Retrieval (IR) , aims to rank documents in response to a user’s query, often without prior knowledge of the user’s specific information need. While transformer-based neural rankers have achieved state-of-the-art performance in ad hoc retrieval, their effectiveness varies significantly across queries. Certain queries—commonly referred to as hard queries —remain particularly challenging, highlighting critical gaps in retrieval models. Identifying these hard queries is essential for improving retrieval systems, motivating the task of Query Performance Prediction (QPP) , which aims to estimate the effectiveness of a query without requiring access to relevance judgments. In this article, we propose Context-aware Query Performance Prediction ( CA-QPP ) , a novel post-retrieval QPP method, which builds on the foundations of perturbation-based QPP methods that hypothesize a relationship between query sensitivity to small perturbations and query retrieval effectiveness. Building on this foundation, our approach exposes the given query to perturbations by constructing two query variations: an effective variation emphasizing terms that enhance retrieval and an ineffective variation accentuating terms that hinder it. By contrasting the retrieval outcomes of these variations using a cross-encoder model, CA-QPP captures the interplay of term contributions and predicts the performance for the given query. We evaluate CA-QPP on the widely used MS MARCO datasets and their associated query sets, including TREC DL 2019 , TREC DL 2020 , DL-Hard , TREC DL 2021 , and TREC DL 2022 , which feature extensive human-labeled relevance judgments. Our experiments demonstrate that CA-QPP consistently outperforms traditional and neural-based QPP baselines across standard correlation metrics, including Pearson’s \(\rho\) , Kendall’s \(\tau\) , and Spearman’s \(\rho\) . Through a detailed case study, we further illustrate the mechanics of CA-QPP and provide empirical evidence for its ability to model the contextual impact of individual query terms, making it a robust framework for query performance prediction.

Recent research has shown that neural information retrieval techniques may be susceptible to adversarial attacks. Adversarial attacks seek to manipulate the ranking of documents, with the intention of exposing users to targeted content. In this article, we introduce the Embedding Perturbation Rank Attack ( EMPRA ) method, a novel approach designed to perform adversarial attacks on black-box Neural Ranking Models (NRMs) . EMPRA manipulates sentence-level embeddings, guiding them towards pertinent context related to the query while preserving semantic integrity. This process generates adversarial texts that seamlessly integrate with the original content and remain imperceptible to humans. Our extensive evaluation conducted on the widely used MS MARCO V1 passage collection as well as the TREC DL 2019 and TREC DL 2020 benchmarks, demonstrate the effectiveness of EMPRA against a wide range of state-of-the-art baselines in promoting a specific set of target documents within a given ranked results. Specifically, on MS MARCO Dev set queries, EMPRA successfully achieves a re-ranking of almost 96% of target documents originally ranked between 51 and 100 to rank within the top 10. Furthermore, EMPRA does not rely on surrogate models for generating adversarial documents, enhancing its robustness against various victim NRMs in realistic settings.

Peer review is foundational to academic publishing, yet the quality of reviews remains difficult to assess at scale due to subjectivity, inconsistency, and the lack of standardized evaluation mechanisms. This talk presents our experience developing and deploying a scalable framework for assessing review quality in operational settings. We combine two complementary approaches: interpretable machine learning models built on quantifiable review- and reviewer-level features, and the application of large language models (LLMs), including Qwen, Phi, and GPT-4o, in zero- and few-shot configurations for textual quality evaluation. We also explore the fine-tuning of LLMs on expert-annotated datasets to examine their upper-bound capabilities. To benchmark these methods, we constructed a dataset of over 700 paper-review pairs labeled by domain experts across multiple quality dimensions. Our findings demonstrate that transparent, feature-based models consistently outperform LLMs in reliability and generalization, particularly when evaluating conceptual depth and argumentative structure. The talk will highlight key engineering choices, deployment challenges, and broader implications for integrating automated review evaluation into scholarly workflows.

The quality of peer review plays a critical role in scientific publishing, yet remains poorly understood and challenging to evaluate at scale. In this work, we introduce RottenReviews, a benchmark designed to facilitate systematic assessment of review quality. RottenReviews comprises over 15,000 submissions from four distinct academic venues enriched with over 9,000 reviewer scholarly profiles and paper metadata. We define and compute a diverse set of quantifiable review-dependent and reviewer-dependent metrics, and compare them against structured assessments from large language models (LLMs) and expert human annotations. Our human-annotated subset includes over 700 paper-review pairs labeled across 13 explainable and conceptual dimensions of review quality. Our empirical findings reveal that LLMs, both zero-shot and fine-tuned, exhibit limited alignment with human expert evaluations of peer review quality. Surprisingly, simple interpretable models trained on quantifiable features outperform fine-tuned LLMs in predicting overall review quality. We publicly release all data, code, and models at https://github.com/Reviewerly-Inc/RottenReviews to support further research in this area.

Entity retrieval plays a critical role in information access systems, yet the development and evaluation of retrieval models remain constrained by the limited availability of high-quality supervision. While recent work has demonstrated the utility of large language models (LLMs) as relevance assessors in passage and document retrieval, their reliability in the context of entity retrieval-where targets are abstract, underspecified, and often semantically sparse-remains unexplored. In this work, we evaluate LLM-based judgments against two complementary supervision signals: human-annotated relevance labels from the DBpedia-Entity benchmark and implicit feedback from user clicks in the LaQuE dataset. We show that LLMs exhibit strong agreement with expert annotations and replicate user click patterns with over 91% agreement, suggesting alignment with behavioral judgments despite noisy input queries. We further identify and analyze systematic mismatches for user clicks on irrelevant entities. Our findings establish LLMs not only as effective annotators for entity relevance judgment-even when given only the entity title-but also as powerful tools for predicting click-through behavior and simulating explainable user intent. Our code, prompts, and data are publicly available at: https://github.com/17shiraz/ClickLLM

This special day event on Responsible Artificial Intelligence (AI) brings together researchers, practitioners, and policymakers to explore how data mining and machine learning systems can be designed to align with ethical principles, societal values, and human well-being. As AI technologies increasingly influence decisions in healthcare, finance, governance, and social systems, there is a critical need to develop frameworks that embed fairness, accountability, and privacy directly into the foundations of knowledge discovery. This full-day event will feature a mix of invited talks, interactive debates, expert panels, and peer-reviewed research presentations, all focused on the practical integration of ethical design into data-driven systems. The Responsible AI Day builds on the success of Canada's NSERC CREATE Program on Responsible AI, an interdisciplinary initiative training the next generation of AI researchers across computer science, law, bioethics, public health, and media studies. Topics will span scalable AI governance, privacy-preserving computation, algorithmic bias mitigation, and the socio-legal tensions emerging in generative AI. By positioning responsible AI as a sociotechnical challenge, this special day aligns with KDD's mission of advancing data science that is not only technically robust but also socially conscious.

Large Language Models (LLMs) exhibit strong capabilities across various Information Retrieval (IR) and natural language processing tasks. However, they are highly sensitive to prompt variations, where slight rephrasings can significantly alter responses, leading to inconsistent or incorrect outputs. This variability poses challenges for response reliability in real-world applications. Inspired by Query Performance Prediction (QPP) in IR, we focus on Prompt Performance Prediction (PPP), which estimates whether an LLM will generate a correct response for a given prompt before execution. We propose VAP3 (Variation-Aware Prompt Performance Prediction), a novel pre-generation PPP approach that integrates prompt variations with adversarial training to enhance robustness against trivial modifications and better capture prompt sensitivity. Evaluating VAP3 against LLM-based self-evaluation, QPP-inspired baselines, and supervised classification models on the PromptSET-HotpotQA and PromptSET-TriviaQA datasets, we demonstrate that VAP3 consistently outperforms all baselines, achieving stable and reliable performance across datasets.

We introduce ROKSANA, an open-source Python toolkit designed to support research in graph-based keyword search under adversarial settings.ROKSANA provides a modular environment for dataset handling, graph neural network (GNN)-based retrieval, and adversarial attack modeling, enabling systematic evaluation of search robustness.The framework integrates built-in retrieval and attack methods while allowing seamless customization of search algorithms and perturbation strategies.Users can benchmark performance on a centralized leaderboard, generate reproducible evaluation reports, and explore ranking behaviors through an interactive web-based visualization interface.By centering around reproducibility, extensibility, and collaborative benchmarking, ROKSANA serves as a comprehensive platform for advancing robust and interpretable keyword search in graphs.This demonstration will showcase ROKSANA's capabilities in real-time, illustrating its impact on experimental workflows and adversarial robustness analysis in graph IR research.

Neural ranking models are widely used to retrieve and rank relevant documents. However, these models may inherit and amplify biases present in the training data, posing challenges for fairness and relevance in ranking outputs. In this paper, we propose a novel curriculum-based training approach that manages bias exposure throughout the training process. We design a bias-aware curriculum that stages the exposure of the model to biased samples during the training stages, allowing the model to establish a fair relevance baseline. We conduct extensive experiments across different LLMs and datasets to evaluate the effectiveness of our approach. Our results demonstrate that our proposed strategy outperforms other bias reduction methods in terms of both fairness and relevance, without sacrificing retrieval effectiveness.

Neural rankers have achieved strong retrieval effectiveness but require large amounts of labeled data, limiting their applicability in few-shot settings. In this paper, we address the sample inefficiency of neural ranking methods by introducing a Reinforcement Learning (RL)-based re-ranking model that achieves high effectiveness with minimal training data. Built on a Deep Q-learning Network (DQN) framework, our approach is designed for few-shot settings, maximizing sample efficiency to ensure robust generalization from limited interactions. Extensive experiments show that our model significantly outperforms data-intensive methods and existing few-shot baselines, demonstrating RL's potential to enhance IR capabilities in few-shot scenarios.

Query performance prediction (QPP) is a key task in information retrieval (IR), focusing on estimating the retrieval quality of a given query without relying on human-labeled relevance judgments. Over the decades, QPP has gained increasing significance, with a surge in research activity in recent years. It has proven to benefit various aspects of retrieval, such as optimizing retrieval effectiveness by selecting the most appropriate ranking function for each query.

The peer review process is a fundamental aspect of academic publishing, ensuring the quality and credibility of scholarly work. In this talk, we will explore the critical challenges associated specifically with the assignment of reviewers to submitted papers. We will introduce Reviewerly, our innovative solution designed to enhance the efficiency and effectiveness of reviewer assignments by leveraging data from diverse sources, including OpenAlex, PubMed, and DBLP. By modeling the reviewer assignment problem as an information retrieval task, we focus on retrieving a pool of relevant and diverse reviewers for each paper.

Prior research has demonstrated that reformulation of queries can significantly enhance retrieval effectiveness. Despite notable successes in neural-based query reformulation methods, identifying optimal reformulations that cover the same information need while enhancing retrieval effectiveness is still challenging. This paper introduces a two-step query reformulation framework for generating and selecting optimal target query variants which not only achieve higher retrieval performance but also preserve the original query's information need. Our comprehensive evaluations on the MS MARCO dataset and TREC Deep Learning tracks demonstrate substantial improvements over original query's performance.

Predicting users’ interests on social networks is gaining attention due to its potential to cater customized information and services to the end users. Although previous works have extensively explored how users’ interests can be modeled on social networks, there has been limited investigation into the prediction of users’ future interests. The objective of our work in this paper is to empirically study the effectiveness of different sets of features based on users’ past social interactions, historical interests and their temporal dynamics to predict their interests over a collection of future-yet-unobserved topics. More specifically, we introduce and formalize the features for interest prediction in four categories: user-based, topical, explicit user-topic engagement, and friends’ influence. We further explore the influence of temporality by augmenting features with information pertaining to users’ historical interests and social connections. We model the task of future interest prediction as a learning-to-rank problem where different features and their related categories are ranked based on their relevance and performance in interest prediction, and investigate the efficiency of different features individually and comparatively for predicting the future interest of users with different activity levels in social networks over on unobserved topics. After conducting experiments on a real-world dataset sourced from Twitter, we have identified several noteworthy findings: (1) relevance feature in the category of past explicit user-topic engagement is the strongest indicator for predicting user’s future interest across all user groups, with an observed 8.57% decrease in NDCG and an 8.95% decrease in MAP when it is removed in the ablation study. (2) the observation of an 8.06% decrease in NDCG and a 7.3% decrease in MAP, when topical features such as popularity, freshness, and coherence are removed in the ablation study, highlights their significance as among the strongest indicators for users’ future interest, particularly for low-active users. (3) although temporal features show a clear positive impact across user groups with varying levels of activity (resulting in a 4.5% decrease in NDCG and a 7.3% decrease in MAP when removed in the ablation study), the temporal topical features do not demonstrate a significant positive effect, and 4) The removal of user-specific characteristics such as influence and personality traits in the ablation study reveals their significant impact in predicting future interest over cold topics, reflected by a 5.49% decrease in NDCG and a 5.72% decrease in MAP. Our findings make significant contributions to the field of future interest prediction, offering valuable insights and practical implications for various applications in social network analysis.

Researchers have already observed social contagion effects in both in-person and online interactions. However, such studies have primarily focused on users’ beliefs, mental states, and interests. In this article, we expand the state of the art by exploring the impact of social contagion on social alignment, i.e., whether the decision to socially align oneself with the general opinion of the users on the social network is contagious to one’s connections on the network or not. The novelty of our work in this article includes: 1) unlike earlier work, this article is among the first to explore the contagiousness of the concept of social alignment on social networks; 2) our work adopts an instrumental variable approach to determine reliable causal relations between observed social contagion effects on the social network; and 3) our work expands beyond the mere presence of contagion in social alignment and also explores the role of population heterogeneity on social alignment contagion. Based on the systematic collection and analysis of data from two large social network platforms, namely, Twitter and Foursquare, we find that a user’s decision to socially align or distance from social topics and sentiments influences the social alignment decisions of their connections on the social network. We further find that such social alignment decisions are significantly impacted by population heterogeneity.

Team formation is concerned with the identification of a group of experts who have a high likelihood of effectively collaborating with each other to satisfy a collection of input skills. Solutions to this task have mainly adopted graph operations and at least have the following limitations: (1) they are computationally demanding, as they require finding shortest paths on large collaboration networks; (2) they use various types of heuristics to reduce the exploration space over the collaboration network to become practically feasible; therefore, their results are not necessarily optimal; and (3) they are not well-suited for collaboration network structures given the sparsity of these networks. Our work proposes a variational Bayesian neural network architecture that learns representations for teams whose members have collaborated with each other in the past. The learned representations allow our proposed approach to mine teams that have a past collaborative history and collectively cover the requested desirable set of skills. Through our experiments, we demonstrate that our approach shows stronger performance compared to a range of strong team formation techniques from both quantitative and qualitative perspectives.

Estimating query difficulty, also known as Query Performance Prediction (QPP), is concerned with assessing the retrieval quality of a ranking method for an input query. Most traditional unsupervised frequency-based models and many recent supervised neural methods have been designed specifically for predicting the performance of sparse retrievers such as BM25. In this paper we propose an unsupervised QPP method for dense neural retrievers which operates by redefining the well-known concept of query robustness i.e., a more robust query to perturbations is an easier query to handle. We propose to generate query perturbations for measuring query robustness by systematically injecting noise into the contextualized neural representation of each query. We then compare the retrieved list for the original query with that of the perturbed query as a way to measure query robustness. Our experiments on four different query sets including MS MARCO, TREC Deep Learning track 2019 and 2020 and TREC DL-Hard show consistently improved performance on linear and ranking correlation metrics over the state of the art.

Researchers have shown that the retrieval effectiveness of queries may depend on other factors in addition to the semantics of the query. In other words, several queries expressed with the same intent, and even using overlapping keywords, may exhibit completely different degrees of retrieval effectiveness. As such, the objective of our work in this paper is to propose a neural disentanglement method that is able to disentangle query semantics from query difficulty. The disentangled query semantics representation provides the means to determine semantic association between queries whereas the disentangled query difficulty representation would allow for the estimation of query effectiveness. We show through our experiments on the query performance prediction; and, query similarity calculation tasks that our proposed disentanglement method is able to show better performance compared to the state of the art.

The information retrieval community has observed significant performance improvements over various tasks due to the introduction of neural architectures. However, such improvements do not necessarily seem to have happened uniformly across a range of queries. As we will empirically show in this paper, the performance of neural rankers follow a long-tail distribution where there are many subsets of queries, which are not effectively satisfied by neural methods. Despite this observation, performance is often reported using standard retrieval metrics, such as MRR or nDCG, which capture average performance over all queries. As such, it is not clear whether reported improvements are due to incremental boost on a small subset of already well-performing queries or addressing queries that have been difficult to address by existing methods. In this paper, we propose the Task Subspace Coverage (TaSC /tAHsk/) metric, which systematically quantifies whether and to what extent improvements in retrieval effectiveness happen on similar or disparate query subspaces for different rankers. Our experiments show that the consideration of our proposed TaSC metric in conjunction with existing ranking metrics provides deeper insight into ranker performance and their contribution to overall advances on a given task.

Network representation learning and its applications have received increasing attention. Due to their various application areas, many research groups have developed a diverse range of software tools and techniques to learn representation for different types of networks. However, to the best of our knowledge, there are limited works that support representation learning for dynamic heterogeneous networks. The work presented in this demonstration paper attempts to fill the gap in this space by developing and publicly releasing an open-source Python library known as, PyDHNet, a Python Library for Dynamic Heterogeneous Network Representation Learning and Evaluation. PyDHNet consists of two main components: dynamic heterogeneous network representation learning and task-specific evaluation. In our paper, we demonstrate that PyDHNet has an extensible architecture, is easy to install (through PIP) and use, and integrates quite seamlessly with other Python libraries. We also show that the implementation for PyDHNet is efficient and enjoys a competitive execution time.

Recent studies have shown that it is possible for stereotypical gender biases to find their way into representational and algorithmic aspects of retrieval methods; hence, exhibit themselves in retrieval outcomes. In this tutorial, we inform the audience of various studies that have systematically reported the presence of stereotypical gender biases in Information Retrieval (IR) systems. We further classify existing work on gender biases in IR systems as being related to (1) relevance judgement datasets, (2) structure of retrieval methods, and (3) representations learnt for queries and documents. We present how each of these components can be impacted by or cause intensified biases during retrieval. Based on these identified issues, we then present a collection of approaches from the literature that have discussed how such biases can be measured, controlled, or mitigated. Additionally, we introduce publicly available datasets that are often used for investigating gender biases in IR systems as well as evaluation methodology adopted for determining the utility of gender bias mitigation strategies.

While neural rankers continue to show notable performance improvements over a wide variety of information retrieval tasks, there have been recent studies that show such rankers may intensify certain stereotypical biases. In this paper, we investigate whether neural rankers introduce retrieval effectiveness (performance) disparities over queries related to different genders. We specifically study whether there are significant performance differences between male and female queries when retrieved by neural rankers. Through our empirical study over the MS MARCO collection, we find that such performance disparities are notable and that the performance disparities may be due to the difference between how queries and their relevant judgements are collected and distributed for different gendered queries. More specifically, we observe that male queries are more closely associated with their relevant documents compared to female queries and hence neural rankers are able to more easily learn associations between male queries and their relevant documents. We show that it is possible to systematically balance relevance judgment collections in order to reduce performance disparity between different gendered queries without negatively compromising overall model performance.

Rapid technological innovations, especially in the information technology space, demand the workforce to be vigilant by acquiring new skills to remain relevant and employable. The workforce needs to be engaged in a continuous lifelong learning process by educating themselves about skills that will be in demand in the future. To do so, it is essential for students, job seekers, and even recruiters to know which skills will be in demand in the future and to invest time and resources in developing these skills. On this basis, the main objective of this paper is to investigate whether social content can offer insight into potential future in-demand skills in the IT job market. Based on the analysis of social content from Reddit and job posting data from Dice and Monster websites, we find that social content related to job skills is a strong indicator for future in-demand skills. We further find that specific social content associated with recruitment-related topics are stronger indicators of future skills. Our findings encourage learners and job seekers to pay close attention to online social content to strategically plan new skills and maximize their employability.

Researchers have already shown that it is possible to improve retrieval effectiveness through the systematic reformulation of users' queries. Traditionally, most query reformulation techniques relied on unsupervised approaches such as query expansion through pseudo-relevance feedback. More recently and with the increasing effectiveness of neural sequence-to-sequence architectures, the problem of query reformulation has been studied as a supervised query translation problem, which learns to rewrite a query into a more effective alternative. While quite effective in practice, such supervised query reformulation methods require a large number of training instances. In this paper, we present three large-scale query reformulation datasets, namely Diamond, Platinum and Gold datasets, based on the queries in the MS MARCO dataset. The Diamond dataset consists of over 188,000 query pairs where the original source query is matched with an alternative query that has a perfect retrieval effectiveness (an average precision of 1). To the best of our knowledge, this is the first set of datasets for supervised query reformulation that offers perfect query reformulations for a large number of queries. The implementation of our fully automated tool, which is based on a transformer architecture, and our three datasets are made publicly available. We also establish a neural query reformulation baseline performance on our datasets by reporting the performance of strong neural query reformulation baselines. It is our belief that our datasets will significantly impact the development of supervised query reformulation methods in the future.

Query Performance Prediction (QPP) is focused on estimating the difficulty of satisfying a user query for a certain retrieval method. While most state of the art QPP methods are based on term frequency and corpus statistics, more recent work in this area have started to explore the utility of pretrained neural embeddings, neural architectures and contextual embeddings. Such approaches extract features from pretrained or contextual embeddings for the sake of training a supervised performance predictor. In this paper, we adopt contextual embeddings to perform performance prediction, but distinguish ourselves from the state of the art by proposing to directly fine-tune a contextual embedding, i.e., BERT, specifically for the task of query performance prediction. As such, our work allows the fine-tuned contextual representations to estimate the performance of a query based on the association between the representation of the query and the retrieved documents. We compare the performance of our approach with the state-of-the-art based on the MS MARCO passage retrieval corpus and its three associated query sets: (1) MS MARCO development set, (2) TREC DL 2019, and (3) TREC DL 2020. We show that our approach not only shows significant improved prediction performance compared to all the state-of-the-art methods, but also, unlike past neural predictors, it shows significantly lower latency, making it possible to use in practice.

During the recent years and with the growing influence of neural architectures, tasks such as ad hoc retrieval have witnessed an impressive improvement in performance. For instance, the performance of rankers on the passage retrieval task on the MS MARCO dataset has improved by an order of magnitude in less than two years. In this paper, we go beyond the overall performance of the state of the art rankers and empirically study their performance from a finer-grained perspective. We find that while neural rankers have been able to consistently improve performance, this has been in part thanks to a specific set of queries from within the larger query set. We systematically show that there are subsets of queries that are difficult for each and every one of the neural rankers, which we refer to as obstinate queries. We show the obstinate queries are similar to easier queries in terms of their number of available relevant judgement documents and the length of the query itself but they are extremely more difficult to satisfy by existing rankers. Furthermore, we observe that query reformulation methods cannot help these queries. On this basis, we present three datasets derived from the MS MARCO Dev set, called the MS MARCO Chameleon datasets. We believe that the next breakthrough in performance would need to necessarily consider the queries in the MS MARCO Chameleons, as such, propose that a well-rounded evaluation strategy for any new ranker would need to include performance measures on both the overall MS MARCO dataset as well as the proposed MS MARCO Chameleon datasets.

The popularity of online social coding (SC) platforms such as GitHub is growing due to their social functionalities and tremendous support during the product development lifecycle. The rich information of experts' contributions on repositories can be leveraged to recruit experts for new/existing projects. In this paper, we define the problem of collaborative experts finding in SC platforms. Given a project, we model an SC platform as an attributed heterogeneous network, learn latent representations of network entities in an end-to-end manner and utilize them to discover collaborative experts to complete a project. Extensive experiments on real-world datasets from GitHub indicate the superiority of the proposed approach over the state-of-the-art in terms of a range of performance measures.

Learning the latent representations of entities based on their relationships and the data associated with them is an essential task in many applications such as ranking, recommendation systems, graph-based team formation, keyword search, and many more. However, the majority of existing techniques learn the latent representations of either network or textual data. Structural embedding techniques suffer from the sparsity of real-world networks. Attributes of nodes are a source of rich information to ameliorate network embedding vectors which are overlooked in the literature. Thus, most existing network representation learning tools capture structural information. This paper introduces an open-source toolkit called OpenAttHetRL to learn the latent representations of entities based on their both network and textual data in an end-to-end fashion. OpenAttHetRL is easy to employ and adapt for a variety of tasks including ranking, recommendation systems, and expert finding. OpenAttHetRL aims to provide a unified toolkit for data pre-processing, building and training models, and performing predictions for a downstream task. It employs a graph convolution network to capture the relationships among entities and a kernel pooling technique to preserve the similarity of their textual data in the embedding space. We use expert finding in community question answering systems to demonstrate how OpenAttHetRL can be trained to get latent representations of questions, their askers, tags, and answerers and find potential answerers of new questions.

We present PyTFL, a library written in Python for the team formation task. In team formation task, the main objective is to form a team of experts given a set of skills. We demonstrate an efficient and well-structured open-source toolkit that can easily be imported into Python. Our toolkit incorporates state-of-the-art approaches for team formation, e.g., neural-based team formation, and supports team formation sub-tasks such as collaboration graph preparation, model training and validation, systematic evaluation based on qualitative and quantitative team metrics, and efficient team formation and prediction. While there are strong research papers on the team formation problem, PyTFL is the first toolkit to be publicly released for this purpose.

Various researchers have recently explored the impact of different types of biases on information retrieval tasks such as ad hoc retrieval and question answering. While the impact of bias needs to be controlled in order to avoid increased prejudices, the literature has often viewed the relationship between increased retrieval utility (effectiveness) and reduced bias as a tradeoff where one can suffer from the other. In this paper, we empirically study this tradeoff and explore whether it would be possible to reduce bias while maintaining similar retrieval utility. We show this would be possible by revising the input query through a bias-aware pseudo-relevance feedback framework. We report our findings based on four widely used TREC corpora namely Robust04, Gov2, ClueWeb09 and ClueWeb12 and using two classes of bias metrics. The findings of this paper are significant as they are among the first to show that decrease in bias does not necessarily need to come at the cost of reduced utility.

Given a set of required skills, the objective of the team formation problem is to form a team of experts that cover the required skills. Most existing approaches are based on graph methods, such as minimum-cost spanning trees. These approaches, due to their limited view of the network, fail to capture complex interactions among experts and are computationally intractable. More recent approaches adopt neural architectures to learn a mapping between the skills and experts space. While they are more effective, these techniques face two main limitations: (1) they consider a fixed representation for both skills and experts, and (2) they overlook the significant amount of past collaboration network information. We learn dense representations for skills and experts based on previous collaborations and bootstrap the training process through transfer learning. We also propose to fine-tune the representation of skills and experts while learning the mapping function. Our experiments over the DBLP dataset verify that our proposed architecture is able to outperform the state-of-the-art graph and neural methods over both ranking and quality metrics.

Abstract Social interactions through online social media have become a daily routine of many, and the number of those whose real world (offline) and online lives have become intertwined is continuously growing. As such, the interplay of individuals' online and offline activities has been the subject of numerous research studies, the majority of which explored the impact of people's online actions on their offline activities. The opposite direction of impact—the effect of real‐world activities on online actions—has also received attention but to a lesser degree. To contribute to the latter form of impact, this paper reports on a quasi‐experimental design study that examined the presence of causal relations between real‐world activities of online social media users and their online emotional expressions. To this end, we have collected a large dataset (over 17K users) from Twitter and Foursquare, and systematically aligned user content on the two social media platforms. Users' Foursquare check‐ins provided information about their offline activities, whereas the users' expressions of emotions and moods were derived from their Twitter posts. Since our study was based on a quasi‐experimental design, to minimize the impact of covariates, we applied an innovative model of computing propensity scores. Our main findings can be summarized as follows: (a) users' offline activities do impact their affective expressions, both of emotions and moods, as evidenced in their online shared textual content; (b) the impact depends on the type of offline activity and if the user embarks on or abandons the activity. Our findings can be used to devise a personalized recommendation mechanism to help people better manage their online emotional expressions.

The speed of digital transformation has resulted in new challenges for job seekers to become lifelong learners and to develop new skills faster than before. In this paper, our main objective is to examine how online content can serve as indicators for changes to the Information Technology (IT) industry and its in-demand skills. To study this relationship, we collect Reddit posts to represent social media content and job postings to reflect the IT industry based on which we explore possible correlations between them. Further, we propose a methodology to quantitatively estimate the predictive power of social media content for future in-demand skills. Our results show that the frequency of skill-related conversations on Reddit correlates with the popularity of skills in job posting data. Additionally, our findings indicate that the number of social posts dedicated to a specific skill can be a strong indicator for future job requirements. This is an important finding because identifying what skills the labor force should acquire will assist job seekers to plan their lifelong learning objectives to (a) maximize their employability, (b) continuously update their skills to remain in demand, and (c) be informed and actively engaged in defining knowledge trends, rather than reactively becoming informed of the latest information.

For the fifth edition of the workshop on Mining Actionable Insights from Social Networks (MAISoN), we organized a special edition with focus on dis/misinformation mining from social media, co-located with CIKM 2020. This topic has attracted a lot of interest from the community since the Coronavirus (COVID-19) epidemic has given rise to an increase of misinformation on social media. The aim of this edition was to bring together researchers from different disciplines interested in mining dis/misinformation on social media. In particular, the distinguishing focus of this special edition was its emphasis on techniques that use social media data for building diagnostic, predictive and prescriptive analysis models related to misinformation. This means that there is rigorous attention for techniques that can be used to understand how and why dis/misinformation is created and spread, to uncover hidden and unexpected aspects of dis/misinformation content, and to recommend insightful countermeasures to restrict the circulation of dis/misinformation and alleviate their negative effects.

We focus on the composition of teams of experts that collectively cover a set of required skills based on their historical collaboration network and expertise. Prior works are primarily based on the shortest path between experts on the expert collaboration network, and suffer from three major shortcomings: (1) they are computationally expensive due to the complexity of finding paths on large network structures; (2) they use a small portion of the entire historical collaboration network to reduce the search space; hence, may form sub-optimal teams; and, (3) they fall short in sparse networks where the majority of the experts have only participated in a few teams in the past. Instead of forming a large network of experts, we propose to learn relationships among experts and skills through a variational Bayes neural architecture wherein: i) we consider all past team compositions as training instances to predict future teams; ii) we bring scalability for large networks of experts due to the neural architecture; and, iii) we address sparsity by incorporating uncertainty on the neural network's parameters which yields a richer representation and more accurate team composition. We empirically demonstrate how our proposed model outperforms the state-of-the-art approaches in terms of effectiveness and efficiency based on a large DBLP dataset.

In this paper, we implement and publicly share a configurable software workflow and a collection of gold standard datasets for training and evaluating supervised query refinement methods. Existing datasets such as AOL and MS MARCO, which have been extensively used in the literature for this purpose, are based on the weak assumption that users' input queries improve gradually within a search session, i.e., the last query where the user ends her information seeking session is the best reconstructed version of her initial query. In practice, such an assumption is not necessarily accurate for a variety of reasons, e.g., topic drift. The objective of our work is to enable researchers to build gold standard query refinement datasets without having to rely on such weak assumptions. Our software workflow, which generates such gold standard query datasets, takes three inputs: (1) a dataset of queries along with their associated relevance judgements (e.g. TREC topics), (2) an information retrieval method (e.g., BM25), and (3) an evaluation metric (e.g., MAP), and outputs a gold standard dataset. The produced gold standard dataset includes a list of revised queries for each query in the input dataset, each of which effectively improves the performance of the specified retrieval method in terms of the desirable evaluation metric. Since our workflow can be used to generate gold standard datasets for any input query set, in this paper, we have generated and publicly shared gold standard datasets for TREC queries associated with Robust04, Gov2, ClueWeb09, and ClueWeb12. The source code of our software workflow, the generated gold datasets, and benchmark results for three state-of-the-art supervised query refinement methods over these datasets are made publicly available for reproducibility purposes.

Social media users readily share their preferences, life events, sentiment and opinions, and implicitly signal their thoughts, feelings, and psychological behavior. This makes social media a viable source of information to accurately and effectively mine users' interests with the hopes of enabling more effective user engagement, better quality delivery of appropriate services and higher user satisfaction. In this tutorial, we cover five important aspects related to the effective mining of user interests: (1) the foundations of social user interest modeling, such as information sources, various types of representation models and temporal features, (2) techniques that have been adopted or proposed for mining user interests, (3) different evaluation methodologies and benchmark datasets, (4) different applications that have been taking advantage of user interest mining from social media platforms, and (5) existing challenges, open research questions and exciting opportunities for further work.

We introduce Vis-Trec, an open-source cross-platform system, which provides the capability to perform in-depth analysis of the results obtained from trec-style evaluation campaigns. Vis-Trec allows researchers to dig deeper in their evaluations by providing various visualizations of the results based on performance percentiles, query difficulty, and comparative analysis of different methods using help-hurt diagrams at the query level. It also automatically organizes the obtained results in tabular LaTeX format that can be used for reporting evaluation findings. The added benefit for Vis-Trec is that it has been developed in Python and is extensible by other developers. The source code along with a functional version of the program are released to the public.

Research in social network analytics has already extensively explored how engagement on online social networks can lead to observable effects on users' real-world behavior (e.g., changing exercising patterns or dietary habits), and their psychological states. The objective of our work in this paper is to investigate the flip-side and examine whether engaging in or disengaging from real-world activities would reflect itself in users' affective processes such as anger, anxiety, and sadness, as expressed in users' posts on online social media. We have collected data from Foursquare and Twitter and found that engaging in or disengaging from a real-world activity, such as frequenting at bars or stopping going to a gym, have direct impact on the users' affective processes. In particular, we report that engaging in a routine real-world activity leads to expressing less emotional content online, whereas the reverse is observed when users abandon a regular real-world activity.

The focus of point-of-interest recommendation techniques is to suggest a venue to a given user that would match the users' interests and is likely to be adopted by the user. Given the multitude of venues and the sparsity of user check-ins, the problem of recommending venues has shown to be a difficult task. Existing literature has already explored various types of features such as geographical distribution, social structure and temporal behavioral patterns to make a recommendation. In this paper, we propose a new set of features derived based on the neural embeddings of venues and users. We show how the neural embeddings for users and venues can be jointly learnt based on the prior check-in sequence of users and then be used to define three types of features, namely user, venue, and user-venue interaction features. These features are integrated into a feature-based matrix factorization model. Our experiments show that the features defined over the user and venue embeddings are effective for venue recommendation.

Specificity is the level of detail at which a given term is represented. Existing approaches to estimating term specificity are primarily dependent on corpus-level frequency statistics. In this work, we explore how neural embeddings can be used to define corpus-independent specificity metrics. Particularly, we propose to measure term specificity based on the distribution of terms in the neighborhood of the given term in the embedding space. The intuition is that a term that is surrounded by other terms in the embedding space is more likely to be specific while a term surrounded by less closely related terms is more likely to be generic. On this basis, we leverage geometric properties between embedded terms to define three groups of metrics: (1) neighborhood-based, (2) graph-based and (3) cluster-based metrics. Moreover, we employ learning-to-rank techniques to estimate term specificity in a supervised approach by employing the three proposed groups of metrics. We curate and publicly share a test collection of term specificity measurements defined based on Wikipedia's category hierarchy. We report on our experiments through metric performance comparison, ablation study and comparison against the state-of-the-art baselines.

he abundance of user generated content on social networks pro-vides the opportunity to build models that are able to accurately and effectively extract, mine and predict users' interests with the hopes of enabling more effective user engagement, better quality delivery of appropriate services and higher user satisfaction. While traditional methods for building user profiles relied on AI-based preference elicitation techniques that could have been considered to be intrusive and undesirable by the users, more recent advances are focused on a non-intrusive yet accurate way of determining users' interests and preferences. In this tutorial, we cover five important aspects related to the effective mining of user interests: (1) we introduce the information sources that are used for extracting user interests, (2) various types of user interest profiles that have been proposed in the literature, (3) techniques that have been adopted or proposed for mining user interests, (4) the scalability and re-source requirements of the state of the art methods, and finally (5)the evaluation methodologies that are adopted in the literature for validating the appropriateness of the mined user interest profiles.We also introduce existing challenges, open research question and exciting opportunities for further work.

The abundance of user generated content on social networks provides the opportunity to build models that are able to accurately and effectively extract, mine and predict users' interests with the hopes of enabling more effective user engagement, better quality delivery of appropriate services and higher user satisfaction. While traditional methods for building user profiles relied on AI-based preference elicitation techniques that could have been considered to be intrusive and undesirable by the users, more recent advances are focused on a non-intrusive yet accurate way of determining users' interests and preferences. In this tutorial, we cover five important aspects related to the effective mining of user interests: (1) we introduce the information sources that are used for extracting user interests, (2) various types of user interest profiles that have been proposed in the literature, (3) techniques that have been adopted or proposed for mining user interests, (4) the scalability and resource requirements of the state of the art methods, and finally (5) the evaluation methodologies that are adopted in the literature for validating the appropriateness of the mined user interest profiles. We also introduce existing challenges, open research question and exciting opportunities for further work.

The systematic linking of explicitly-observed phrases within a document to entities of a knowledge base has already been explored in a process known as entity linking. The objective of this paper, however, is to identify and entity link those entities that are not mentioned but are implied within a document, more specifically within a tweet. This process is referred to as implicit entity linking. Unlike prior work that build a representation for each entity based on its related content in the knowledge base, we propose to perform implicit entity linking by determining how a tweet is related to user-generated content posted online and as such indirectly perform entity linking. We formulate this problem as an ad-hoc document retrieval process where the input query is the tweet, which needs to be implicitly linked and the document space is the set of user-generated content related to the entities of the knowledge base. We systematically compare our work with the state-of-the-art baseline and show that our method is able to provide statistically significant improvements.

The accurate prediction of users' future topics of interests on social networks can facilitate content recommendation and platform engagement. However, researchers have found that future interest prediction, especially on social networks such as Twitter, is quite challenging due to the rapid changes in community topics and evolution of user interactions. In this context, temporal collaborative filtering methods have already been used to perform user interest prediction, which benefit from similar user behavioral patterns over time to predict how a user's interests might evolve in the future. In this paper, we propose that instead of considering the whole user base within a collaborative filtering framework to predict user interests, it is possible to much more accurately predict such interests by only considering the behavioral patterns of the most influential users related to the user of interest. We model influence as a form of causal dependency between users. To this end, we employ the concept of Granger causality to identify causal dependencies. We show through extensive experimentation that the consideration of only one causally dependent user leads to much more accurate prediction of users' future interests in a host of measures including ranking and rating accuracy metrics.

Neural embeddings have been effectively integrated into information retrieval tasks including ad hoc retrieval. One of the benefits of neural embeddings is they allow for the calculation of the similarity between queries and documents through vector similarity calculation methods. While such methods have been effective for document matching, they have an inherent bias towards documents that are sized relatively similarly. Therefore, the difference between the query and document lengths, referred to as the query-document size imbalance problem, becomes an issue when incorporating neural embeddings and their associated similarity calculation models into the ad hoc document retrieval process. In this paper, we propose that document representation methods need to be used to address the size imbalance problem and empirically show their impact on the performance of neural embedding-based ad hoc retrieval. In addition, we explore several types of document representation methods and investigate their impact on the retrieval process. We conduct our experiments on three widely used standard corpora, namely Clueweb09B, Clueweb12B and Robust04 and their associated topics. Summarily, we find that document representation methods are able to effectively address the query-document size imbalance problem and significantly improve the performance of neural ad hoc retrieval. In addition, we find that a document representation method based on a simple term-frequency shows significantly better performance compared to more sophisticated representation methods such as neural composition and aspect-based methods.

Researchers have shown that it is possible to identify reported instances of personal life events from users' social content, e.g., tweets. This is known as personal life event detection. In this paper, we take a step forward and explore the possibility of predicting users' next personal life event based solely on the their historically reported personal life events, a task which we refer to as personal life event prediction. We present a framework for modeling streaming social content for the purpose of personal life event prediction and describe how various instantiations of the framework can be developed to build a life event prediction model. In our extensive experiments, we find that (i) historical personal life events of a user have strong predictive power for determining the user's future life event; (ii) the consideration of sequence in historically reported personal life events shows inferior performance compared to models that do not consider sequence, and (iii) the number of historical life events and the length of the past time intervals that are taken into account for making life event predictions can impact prediction performance whereby more recent life events show more relevance for the prediction of future life events.

User communities in social networks are usually identified by considering explicit structural social connections between users. While such communities can reveal important information about their members such as family or friendship ties and geographical proximity, just to name a few, they do not necessarily succeed at pulling like‐minded users that share the same interests together. Therefore, researchers have explored the topical similarity of social content to build like‐minded communities of users. In this article, following the topic‐based approaches, we are interested in identifying communities of users that share similar topical interests with similar temporal behavior. More specifically, we tackle the problem of identifying temporal (diachronic) topic‐based communities, i.e., communities of users who have a similar temporal inclination toward emerging topics. To do so, we utilize multivariate time series analysis to model the contributions of each user toward emerging topics. Further, our modeling is completely agnostic to the underlying topic detection method. We extract topics of interest by employing seminal topic detection methods; one graph‐based and two latent Dirichlet allocation‐based methods. Through our experiments on Twitter data, we demonstrate the effectiveness of our proposed temporal topic‐based community detection method in the context of news recommendation, user prediction, and document timestamp prediction applications, compared with the nontemporal as well as the state‐of‐the‐art temporal approaches.

Objective: The goal of this work is to map Unified Medical Language System (UMLS) concepts to DBpedia resources using widely accepted ontology relations from the Simple Knowledge Organization System (skos:exactMatch, skos:closeMatch) and from the Resource Description Framework Schema (rdfs:seeAlso), as a result of which a complete mapping from UMLS (UMLS 2016AA) to DBpedia (DBpedia 2015-10) is made publicly available that includes 221 690 skos:exactMatch, 26 276 skos:closeMatch, and 6 784 322 rdfs:seeAlso mappings. Methods: We propose a method called circular resolution that utilizes a combination of semantic annotators to map UMLS concepts to DBpedia resources. A set of annotators annotate definitions of UMLS concepts returning DBpedia resources while another set performs annotation on DBpedia resource abstracts returning UMLS concepts. Our pipeline aligns these 2 sets of annotations to determine appropriate mappings from UMLS to DBpedia. Results: We evaluate our proposed method using structured data from the Wikidata knowledge base as the ground truth, which consists of 4899 already existing UMLS to DBpedia mappings. Our results show an 83% recall with 77% precision-at-one (P@1) in mapping UMLS concepts to DBpedia resources on this testing set. Conclusions: The proposed circular resolution method is a simple yet effective technique for linking UMLS concepts to DBpedia resources. Experiments using Wikidata-based ground truth reveal a high mapping accuracy. In addition to the complete UMLS mapping downloadable in n-triple format, we provide an online browser and a RESTful service to explore the mappings.

This paper explores the possibility of using neural embedding features for enhancing the effectiveness of ad hoc document ranking based on learning to rank models. We have extensively introduced and investigated the effectiveness of features learnt based on word and document embeddings to represent both queries and documents. We employ several learning to rank methods for document ranking using embedding-based features, keyword-based features as well as the interpolation of the embedding-based features with keyword-based features. The results show that embedding features have a synergistic impact on keyword based features and are able to provide statistically significant improvement on harder queries.

We propose a neural embedding approach to identify temporally like-minded user communities, i.e., those communities of users who have similar temporal alignment in their topics of interest. Like-minded user communities in social networks are usually identified by either considering explicit structural connections between users (link analysis), users' topics of interest expressed in their posted contents (content analysis), or in tandem. In such communities, however, the users' rich temporal behavior towards topics of interest is overlooked. Only few recent research efforts consider the time dimension and define like-minded user communities as groups of users who share not only similar topical interests but also similar temporal behavior. Temporal like-minded user communities find application in areas such as recommender systems where relevant items are recommended to the users at the right time. In this paper, we tackle the problem of identifying temporally like-minded user communities by leveraging unsupervised feature learning (embeddings). Specifically, we learn a mapping from the user space to a low-dimensional vector space of features that incorporate both topics of interest and their temporal nature. We demonstrate the efficacy of our proposed approach on a Twitter dataset in the context of three applications: news recommendation, user prediction and community selection, where our work is able to outperform the state-of-the-art on important information retrieval metrics.

This paper addresses the task of document retrieval based on the degree of document relatedness to the meanings of a query by presenting a semantic-enabled language model. Our model relies on the use of semantic linking systems for forming a graph representation of documents and queries, where nodes represent concepts extracted from documents and edges represent semantic relatedness between concepts. Based on this graph, our model adopts a probabilistic reasoning model for calculating the conditional probability of a query concept given values assigned to document concepts. We present an integration framework for interpolating other retrieval systems with the presented model in this paper. Our empirical experiments on a number of TREC collections show that the semantic retrieval has a synergetic impact on the results obtained through state of the art keyword-based approaches, and the consideration of semantic information obtained from entity linking on queries and documents can complement and enhance the performance of other retrieval models.

The first international workshop on Mining Actionable Insights from Social Networks (MAISoN'17) is to be held on February 10, 2017; co-located with the Tenth ACM International Web Search and Data Mining (WSDM) Conference in Cambridge, UK. MAISoN'17 aims at bringing together researchers and participants from different disciplines such as computer science, big data mining, machine learning, social network analysis and other related areas in order to identify challenging problems and share ideas, algorithms, and technologies for mining actionable insight from social network data. We organized a workshop program that includes the presentation of eight peer-reviewed papers and keynote talks, which foster discussions around state-of-the-art in social network mining and will hopefully lead to future collaborations and exchanges.

Runtime adaptive systems are able to dynamically transform their internal structure, and hence their behavior, in response to internal or external changes. Such transformations provide the basis for new functionalities or improvements of the non-functional properties that match operational requirements and standards. Software Product Line Engineering (SPLE) has introduced several models and mechanisms for variability modeling and management. Dynamic software product lines (DSPL) engineering exploits the knowledge acquired in SPLE to develop systems that can be context-aware, post-deployment reconfigurable, or runtime adaptive. This paper focuses on DSPL engineering approaches for developing runtime adaptive systems and proposes a framework for classifying and comparing these approaches from two distinct perspectives: adaptation properties and adaptation realization. These two perspectives are linked together by a series of guidelines that help to select a suitable adaptation realization approach based on desired adaptation types.

Software product line engineering is a discipline that facilitates a systematic reuse-based approach by formally representing commonalities and variabilities between the applications of a target domain. As one of the main artifacts of the software product line, a feature model represents the possible configuration space and can be customized based on the stakeholders' needs. Considering the complexity of the variabilities represented by feature models and the diversity of the stakeholders' expectations, the configuration process can be viewed as a complex optimization problem. In previous research, researchers have bridged the gap between requirement and product line engineering by integrating feature models and goal models. In this paper, we propose an approach for the configuration process that seeks to satisfy the stakeholders' requirements as well as the feature models' structural and integrity constraints. We model stakeholders' functional and nonfunctional needs and their preferences using requirement engineering goal models. We formalize the structure of the feature model, the stakeholders' objectives, and their preferences in the form of an integer linear program to conduct a semi-automated feature model configuration process. Our experimental results show that the proposed configuration framework is scalable when considering both functional and nonfunctional requirements of stakeholders.

Existing work in the semantic relatedness literature has already considered various information sources such as WordNet, Wikipedia and Web search engines to identify the semantic relatedness between two words. We will show that existing semantic relatedness measures might not be directly applicable to microblogging content such as tweets due to i) the informality and short length of microblogging content, which can lead to shift in the meaning of words when used in microblog posts, ii) the presence of non-dictionary words that have their semantics defined/evolved by the Twitter community. Therefore, we propose the Twitter Space Semantic Relatedness (TSSR) technique that relies on the latent relation hypothesis to measure semantic relatedness of words on Twitter. We construct a graph representation of terms in tweets and apply a random walk procedure to produce a stationary distribution for each word, which is the basis for relatedness calculation. Our experiments examine TSSR from three different perspectives and show that TSSR is better suited for Twitter analytics compared to the standard semantic relatedness techniques.

Mining security events helps with better precautionary planning for community safety.However, incident records are expressed in diverse and application dependent formats which impedes common comprehension for automatic knowledge extraction and reasoning.In this paper, we present Security Incident Ontology, SIO, a novel light-weight domain ontology for security incidents.We use Timeline to annotate the temporal facts of incidents and adopt Event to represent any security issues from indecent behavior to assault to more adverse crime which raise the security alarm in a community.It will present a unique way to the security incident detectors, a police officer, Robocops, or intelligent CCTV cameras, to report security events.We use SIO in populating security incident notifications of Integrated Risk Management (IRM) at Ryerson University to evaluate its competency, for Ryerson University campus has both business and housing area in the vicinity and encompass not only high rate, but also wide variety of different security issues.SIO is developed in OWL 2 with Protégé.

Abstract The combination of Agile methods and distributed software development via remote teams represents an emerging approach to address the challenges such as late feedback, slow project timelines, and high cost, typically associated with software development projects. However, when projects are implemented using an Agile model with distributed human resources, there are a number of challenges that need to be considered and mitigated. The objectives of our work are multifold. First, we would like to understand the reasons and conditions that lead to the adoption of distributed Agile software engineering (DASE) practices. Second, we would like to investigate and find out the most important risks that threaten a DASE approach and what mitigation strategies exist to address them. Finally, we would like to highlight which of the available approaches among the existing Agile methodologies has been successfully adopted by the community. We intend to solidify our findings by exploring the strength of the evidence that has been reported in the literature. We carried out a systematic literature review of DASE techniques and approaches. This systematic literature review found time zone difference, knowledge of resources, lack of infrastructure, missing roles, and responsibilities as being the primary challenges that needed to be addressed. Copyright © 2015 John Wiley & Sons, Ltd.

The concept of social goals refers to organizational goals that are defined in an open and transparent manner; they serve as social objects that incite both formal and informal collaboration around shared interests/objectives. Our objective is to facilitate the comprehension of social goals and examine the role of social goals as scaffolds of social learning in an organization. To this end, we followed an approach based on the visualization of social goals and explored how different presentations of goals, specifically, faceted goal browsing, graph-based visualization, and timeline-based visualization, contribute to the realization of the stated objective. To assess this approach, we conducted a between subjects study where each participant performed a set of goal comprehension tasks with one of the examined presentations of goals. The study demonstrated that our visualizations of goals increase the accuracy of the overall comprehension of an organization's goals; this positive effect is also present when the comprehension of relationships-either explicit or implicit ties-among social goals is needed. The results also confirmed that our graph-based visualization of social goals could serve as a facilitator of social learning in an organization.

Software Product Line (SPL) engineering is a systematic reuse-based software development approach which is founded on the idea of building software products using a set of core assets rather than developing individual software systems from scratch. Feature models are among the widely used artefacts for SPL development that mostly capture functional and operational variability of a system. Researchers have argued that connecting intentional variability models such as goal models with feature variability models in a target domain can enrich feature models with valuable quality and non-functional information. Interrelating goal models and feature models has already been proposed in the literature for capturing non-functional properties in software product lines; however, this manual integration process is cumbersome and tedious. In this paper, we propose a (semi) automated approach that systematically integrates feature models and goal models through standard ontologies. Our proposed approach connects feature model and goal model elements through measuring the semantic similarity of their annotated ontological concepts. Our work not only provides the means to systematically interrelate feature models and goal models but also allows domain engineers to identify and model the role and significance of non-functional properties in the domain represented by the feature model.

Self-adaptive systems are a class of software applications, which are able to dynamically transform their internal structure and hence their behavior in response to internal or external stimuli. The transformation may provide the basis for new functionalities or improve or maintain non-functional properties in order to match the application better to its operational requirements and standards. Software Product Line Engineering has rich methods and techniques in variability modeling and management which is one of the main issues in developing self-adaptive systems. Dynamic software product lines (DSPL) have been proposed to exploit the knowledge acquired in SPLE to develop self-adaptive software systems.

Requirements engineering is recognized as a critical stage in software development lifecycle. Given the nature of Software Product Lines (SPL), the importance of requirements engineering is more pronounced as SPLs pose more complex challenges than development of a 'single' product. Several methods have been proposed in the literature, which encompass activities for capturing requirements, their variability and commonality. To investigate the maturity and effectiveness of the current requirements engineering approaches in software product lines, we develop an evaluation framework containing a set of evaluation criteria and assess feature oriented requirements engineering methods based on the proposed criteria. As a result of this initial study, we find out the majority of approaches lacks proper techniques for supporting the validation of family requirements models as well as dealing with delta requirements. Additionally, capturing stakeholders' preferences and applying them during the course of software feature configuration have not been taken into account and addressed in the proposed approaches.

Feature modeling is one of the main techniques used in Software Product Line Engineering to manage the variability within the products of a family. Concrete products of the family can be generated through a configuration process. The configuration process selects and/or removes features from the feature model according to the stakeholders' requirements. Selecting the right set of features for one product from amongst all of the available features in the feature model is a complex task because: 1) the multiplicity of stakeholders' functional requirements; 2) the positive or negative impact of features on non-functional properties; and 3) the stakeholders' preferences w.r.t. the desirable non-functional properties of the final product. Many configurations techniques have already been proposed to facilitate automated product derivation. However, most of the current proposals are not designed to consider stakeholders' preferences and constraints especially with regard to non-functional properties. We address the software product line configuration problem and propose a framework, which employs an artificial intelligence planning technique to automatically select suitable features that satisfy both the stakeholders' functional and non-functional preferences and constraints. We also provide tooling support to facilitate the use of our framework. Our experiments show that despite the complexity involved with the simultaneous consideration of both functional and non-functional properties our configuration technique is scalable.

SUMMARY Feature modeling an attractive technique for capturing commonality as well as variability within an application domain for generative programming and software product line engineering. Feature models symbolize an overarching representation of the possible application configuration space, and can hence be customized based on specific domain requirements and stakeholder goals. Most interactive or semi‐automated feature model customization processes neglect the need to have a holistic approach towards the integration and satisfaction of the stakeholder's soft and hard constraints, and the application‐domain integrity constraints. In this paper, we will show how the structure and constraints of a feature model can be modeled uniformly through Propositional Logic extended with concrete domains, called Pscr (𝒩). Furthermore, we formalize the representation of soft constraints in fuzzy 𝒫(𝒩) and explain how semi‐automated feature model customization is performed in this setting. The model configuration derivation process that we propose respects the soundness and completeness properties. Copyright © 2011 John Wiley & Sons, Ltd.

The six papers in this special issue depict the state of the art and practice of the impact of semantic technologies in the field of Software Engineering.

In Software Product Line Engineering, concrete products of a family can be generated through a configuration process over a feature model. The configuration process selects features from the feature model according to the stakeholders' requirements. Selecting the right set of features for one product from all the available features in the feature model is a cumbersome task because 1) the stakeholders may have diverse business concerns and limited resources that they can spend on a product and 2) features may have negative and positive contributions on different business concern. Many configurations techniques have been proposed to facilitate software developers' tasks through automated product derivation. However, most of the current proposals for automatic configuration are not devised to cope with business oriented requirements and stakeholders' resource limitations. We propose a framework, which employs an artificial intelligence planning technique to automatically select suitable features that satisfy the stakeholders' business concerns and resource limitations. We also provide tooling support to facilitate the use of our framework.

Since the introduction in the early nineties, feature models receive a great deal of attention in industry and academia. Industrial success stories in applying feature models for modeling software product lines, and using them for configuring software-intensive systems motivate academia to discover ways to integrate different feature dependencies into the feature model, and automate verified feature configuration. In this paper we demonstrate how ontologies and Semantic Web technologies facilitate seamless integration of required external services and deployment platform capabilities into the feature model. Furthermore, we also contribute with an algorithm for automating staged configuration using Semantic Web reasoners to discover unfeasible features of the feature model.

Belief merging is concerned with the integration of several belief bases such that a coherent belief base is developed. Various belief merging models that use a belief negotiation game have been developed. These models often consist of two key functions, namely, negotiation and weakening. A negotiation function finds and selects the weakest belief bases among the available belief bases, while the weakening function removes the least valuable set of information from the selected belief base. This process is iteratively repeated until a consistent belief base is developed. In this paper, we extend the current game-based belief merging models by introducing the Subjective belief game model. The Subjective belief game model operates over a Subjective belief profile, which consists of belief bases with Subjectively annotated formulas. The Subjective information attached to each formula enables the proposed model to prioritize the formulas in the merging process. One of the advantages of the proposed game is that it provides room for enhancing the content of the weak belief bases, instead of enforcing their further weakening. Trustworthiness of the information sources is also considered. We provide several instantiations of the model. The Subjective belief game model can be useful for formalizing a negotiation process between the human participants of a design process in cases where discrepancies and conflicts arise. We apply this belief game model to an example case study of collaboratively designing some parts of unified modeling language (UML) class diagram for vehicle design.

During the last decade, anomaly detection has attracted the attention of many researchers to overcome the weakness of signature-based IDSs in detecting novel attacks, and KDDCUP'99 is the mostly widely used data set for the evaluation of these systems. Having conducted a statistical analysis on this data set, we found two important issues which highly affects the performance of evaluated systems, and results in a very poor evaluation of anomaly detection approaches. To solve these issues, we have proposed a new data set, NSL-KDD, which consists of selected records of the complete KDD data set and does not suffer from any of mentioned shortcomings.

The intention of this paper is to introduce a risk analysis methodology called Astrolabe. Astrolabe is based on causal analysis of systems risks. It allows the analysts to both align the current standpoint of the system with its intentions and identify any vulnerabilities or hazards that threaten the systems stability. Astrolabe adopts concepts from organizational theory and software requirement engineering. The aim of Astrolabe is to guide risk analysis through its phases so that a near complete investigation of system risks is performed. The concepts and methods driving the Astrolabe methodology have been clearly explained in this paper.

Impact analysis methods are commonly employed to reduce the likelihood of encountering faulty or unexpected behavior from a software program as a result of developers' oblivious modifications. In this paper, we propose a static impact analysis technique that creates clusters of closely associated software program files based on their co-modification history in the software repository. The proposed method benefits from dimensionality reduction techniques to reduce the complexity of the collected information and perform the impact analysis process faster. The method has been tested on four different open source project repositories, namely Firefox, Firebird, Thunderbird, and FileZilla. The results of the impact analysis method performance in terms of precision (impact set identification accuracy) and execution time cost have been reported in this paper. The proposed method shows promising behavior when used with several specific clustering techniques such as DBscan and X-Means.

Genetic Algorithms are very powerful search methods that are used in different optimization problems. Parallel versions of genetic algorithms are easily implemented and usually increase algorithm performance [4]. Fuzzy control as another optimization solution along with genetic algorithms can significantly increase algorithm performance. Two variations for genetic algorithm and fuzzy system composition exist. In the first approach Genetic algorithms are used to optimize and model the structure of fuzzy systems through knowledge base or membership function design while the second approach exploits fuzzy to dynamically supervise genetic algorithm performance by speedily reaching an optimal solution. In this paper we propose a new method for fuzzy parallel genetic algorithms, in which a parallel client-server single population fuzzy genetic algorithm is configured to optimize the performance of the first three Dejong functions in order to reach a global solution in the least possible iterations. Simulations show much improvement in genetic algorithm performance evaluation.

Clustering belongs to the set of mathematical problems which aim at classification of data or objects into related sets or classes. Many different pattern clustering approaches based on the pattern membership model could be used to classify objects within various classes. Different models of Crisp, Hierarchical, Overlapping and Fuzzy clustering algorithms have been developed which serve different purposes. The main deficiency that most of the algorithms face is that the number of clusters for reaching the optimal arrangement is not automatically calculated and needs user intervention. In this paper we propose a fuzzy clustering technique (FACT) which determines the number of appropriate clusters based on the pattern essence. Different experiments for algorithm evaluation were performed which show a much better performance compared with the typical widely used K-means clustering algorithm

Reputation is a distributed, socially ascribed, and collective belief of the society towards the stand point of a single person, group, role or even a non-human identity within the context of that society. Therefore, reputation can be only formalized based on the underlying principals and values of a specific context. In this paper we propose a model that clearly depicts how the reputation of a person in one context can affect his reputation in other contexts. This model provides a reputation propagation scheme that allows us to analyze the overall behavior of a person within the scope of a multi-context environment. It also caters suitable mechanisms to anticipate a proper initial reputation value for a person within the contexts that he has not been present in before.

Infrastructures are networks of highly complex systems that can be classified as socio-technical organisms with hidden consciousness. The hidden consciousness of these types of systems lies beyond their definition. Although these systems are structurally independent of any outside component, but collaborate synergistically to provide their services to the end customer. The interdependencies between these complex systems bring about sophisticated and unpredictable outcomes. In this paper we propose a platform independent metamodel for critical infrastructures. The metamodel precisely defines every aspect of an infrastructure through clear syntactical and semantic definition of existing concepts and relationships. The Platform independent model (PIM) has been defined as a UML profile (UML-CI) and serves as one of the first steps towards building an agent based simulation environment.