The process of associating elements of learning content with concepts or skills that this content helps students to master is one of the critical steps in developing personalized educational systems. When these associations are properly established, the system can infer the growth of student understanding of separate knowledge components from the logs of their interactions with associated learning content and use it to adapt the learning process accordingly by targeting gaps in individual students’ knowledge. Unfortunately, crafting these links between learning content and knowledge components is a very time- and expertise-demanding process that has traditionally been performed manually by domain experts with the help of knowledge engineers. Recently, the power of Large Language Models has motivated a new generation of research on concept extraction from textual learning content. The work presented in this paper contributes to this trend while introducing two important innovations. First, our concept extraction process is guided by a human-authored ontology of the target domain - Python programming. Second, alongside a traditional expert evaluation of the concept extraction quality, we apply two additional validation approaches: one based on using an educational data mining technique (learning curves) and another utilizing the pedagogical expertise of teaching the target domain (learning content placement).

Adaptive programming practice often relies on fixed libraries of worked examples and practice problems, which require substantial authoring effort and may not correspond well to the logical errors and partial solutions students produce while writing code. As a result, students may receive learning content that does not directly address the concepts they are working to understand, while instructors must either invest additional effort in expanding content libraries or accept a coarse level of personalization. We present an approach for knowledge-component (KC) guided educational content generation using pattern-based KCs extracted from student code. Given a problem statement and student submissions, our pipeline extracts recurring structural KC patterns from students' code through AST-based analysis and uses them to condition a generative model. In this study, we apply this approach to worked example generation, and compare baseline and KC-conditioned outputs through expert evaluation. Results suggest that KC-conditioned generation improves topical focus and relevance to students' underlying logical errors, providing evidence that KC-based steering of generative models can support personalized learning at scale.

Large Language Models (LLMs) are increasingly integrated into CS education, with students using them to generate problem solutions. Although LLMs can produce correct solutions, do their approaches align with the diversity of approaches in student code? This work aims to propose an analytical workflow for investigating this topic. We proposed a workflow and evaluated it using 999 student Java code submissions from two questions in a CS1 dataset, along with matched zero-shot solutions generated by OpenAI GPT-4. Zero-shot prompting was chosen to mirror students' typical use of LLMs. Our preliminary results indicate that students consistently demonstrated greater diversity in problem-solving methods than the LLM, especially on more complex tasks. These findings reinforce concerns that heavy reliance on LLMs might reduce students' creativity when solving programming tasks. They also inspire design implications across several future directions, including cheating detection, the use of simulated students, and pedagogy refinement to promote more diverse approaches to students.

In this paper, we present an approach that enables a broad re-use of English-authored smart learning content by translating it into other languages. To make it possible, we integrated translation functionalities directly into a smart content authoring system and engaging human-AI collaboration. The approach has been used to translate a large volume of worked examples and completion problems in Python from English to Spanish. The translated content has been piloted in several universities in a Spanish-speaking country.

We present a reproducible pipeline that aligns CS1 textbook sections with problems from a public dataset via a Knowledge Component (KC) -a single conceptual skill required for problem solving- ontology. It assigns KCs to sections and problems, respects the prerequisite order to avoid inserting problems too early, and generates tips for not-yet-taught concepts. We evaluate three KC assignment strategies: embedding-only, embedding with a Large Language Model (LLM) tie-breaker, and direct LLM assignment. We find direct assignment matches or exceeds human annotators. Our results show that constrained LLMs can enrich CS1 textbooks with curriculum-aware practice problems.

Despite the wide-spread use of multi-list or carousel (Netflix-like) interfaces in e-commerce and streaming services, there is little academic research (less than 30 papers), especially when compared to works for single-list interfaces. Recent eye tracking results [5] have shown that users browse multi-list and carousels significantly differently than other interfaces. Carousels are much more complex, allowing a wide-range of browsing/interaction sequences with multiple topic defined-lists that can be swiped to see more items. To account for this complexity and improve recommendations, recommender systems should be designed specifically for the interfaces they are used on, in other words interface-aware recommenders.

Jan Cegin, Jakub Simko, Peter Brusilovsky. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2025.

The 12th Joint Workshop on Interfaces and Human Decision Making for Recommender Systems (IntRS'25) takes a user-centric perspective on recommender systems research. It brings together an interdisciplinary community of researchers and practitioners who explore a range of important issues on the "user side"of recommender systems, including user studies, psychology-informed design of RSs, novel interfaces, and evaluation methodologies. Its purpose is to identify critical challenges and emerging topics with a strong focus on fundamental historical challenges in the field. In this summary, we introduce the motivation and perspective of the workshop, review its history, and discuss the most critical issues that deserve attention for future research directions.

Novice programmers can greatly improve their understanding of challenging programming concepts by studying worked examples that demonstrate the implementation of these concepts. Despite the extensive repositories of effective worked examples created by CS education experts, a key challenge remains: identifying the most relevant worked example for a given programming problem and the specific difficulties a student faces solving the problem. Previous studies have explored similar example recommendation approaches. Our research introduces a novel method by utilizing deep learning code representation models to generate code vectors, capturing both syntactic and semantic similarities among programming examples. Driven by the need to provide relevant and personalized examples to programming students, our approach emphasizes similarity assessment and clustering techniques to identify similar code problems, examples, and challenges. This method aims to deliver more accurate and contextually relevant recommendations based on individual learning needs. Providing tailored support to students in real-time facilitates better problem-solving strategies and enhances students' learning experiences, contributing to the advancement of programming education.

Knowledge tracing is a method to model students' knowledge and enable personalized education in many STEM disciplines such as mathematics and physics, but has so far still been a challenging task in computing disciplines.One key obstacle to successful knowledge tracing in computing education lies in the accurate extraction of knowledge components (KCs), since multiple intertwined KCs are practiced at the same time for programming problems.In this paper, we address the limitations of current methods and explore a hybrid approach for KC extraction, which combines automated code parsing with an expert-built ontology.We use an introductory (CS1) Java benchmark dataset to compare its KC extraction performance with the traditional extraction methods using a state-of-the-art evaluation approach based on learning curves.Our preliminary results show considerable improvement over traditional methods of student modeling.The results indicate the opportunity to improve automated KC extraction in CS education by incorporating expert knowledge into the process.

Generative artificial intelligence (GenAI) tools, such as large language models (LLMs), generate natural language and other types of content to perform a wide range of tasks. This represents a significant technological advancement that poses opportunities and challenges to educational research and practice. This commentary brings together contributions from nine experts working in the intersection of learning and technology and presents critical reflections on the opportunities, challenges, and implications related to GenAI technologies in the context of education. In the commentary, it is acknowledged that GenAI’s capabilities can enhance some teaching and learning practices, such as learning design, regulation of learning, automated content, feedback, and assessment. Nevertheless, we also highlight its limitations, potential disruptions, ethical consequences, and potential misuses. The identified avenues for further research include the development of new insights into the roles human experts can play, strong and continuous evidence, human-centric design of technology, necessary policy, and support and competence mechanisms. Overall, we concur with the general skeptical optimism about the use of GenAI tools such as LLMs in education. Moreover, we highlight the danger of hastily adopting GenAI tools in education without deep consideration of the efficacy, ecosystem-level implications, ethics, and pedagogical soundness of such practices.

Jan Cegin, Branislav Pecher, Jakub Simko, Ivan Srba, Maria Bielikova, Peter Brusilovsky. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.

The primary goal of Recommender Systems is to suggest the most suitable items to a user, aligning them with the user’s interests and needs. RSs are essential for modern e-commerce, helping users discover content and products by predicting suitable items based on their past behavior. However, their success isn’t just about advanced algorithms. The design of the user interface and a good integration with the human decision-making process are equally crucial. A well-designed interface enhances the user experience and makes recommendations more effective, while a poor interface can lead to frustration. Recognizing this limitation, recent trends in Recommender Systems (RSs) are increasingly focusing on integrating Symbiotic Human-Machine Decision-Making models. These models aim to offer users a dynamic and persuasive interface that helps them better understand and engage with recommendations. This shift is a crucial step toward developing recommender systems that truly connect with users and offer a more enjoyable, trustworthy, explainable, and user-friendly experience. Although early efforts concentrated on creating systems that could proactively predict user preferences and needs, modern RSs also emphasize the importance of providing users with control and transparency over their recommendations. Finding the right balance between proactivity and user control is essential to ensure that the system supports users without being too intrusive, thus improving their overall satisfaction. As Large Language Models (LLMs) become more integrated into recommender systems, the importance of user-centric interfaces and a deep understanding of decision-making becomes even more critical. Effective integration of LLMs requires interfaces that are both visually and cognitively engaging.

Many SIGCSE attendees are either developing or using online educational tools, and all will benefit from better interoperability among these tools and better analysis of the clickstream data coming from those tools. New tools for analyzing big data leveraged by AI (e.g., deep learning for assessment) in turn improve both content and pedagogy, thus setting up a virtuous cycle fueling learning discoveries and leveraging innovation in AI: Online technologies → big data analysis → better online technologies. This NSF-supported workshop is the latest in a series of SPLICE workshops, and is a continuation of our event at SIGCSE 2023, where the SPLICE-Portal, a dedicated socio-technical research infrastructure for Computing Education Research, was presented. This year, we continue the work with several new SPLICE community working groups, including those on Dashboards, Large Language Models, Parsons Problems, and Smart Learning Content Protocols. We continue to build upon our existing collaborations developed over the course of the project to engage more members of the community in tasks that will advance the project agenda.

Offline data-driven evaluation is considered a low-cost and more accessible alternative to the online empirical method of assessing the quality of recommender systems. Despite their popularity and effectiveness, most data-driven approaches are unsuitable for evaluating interactive recommender systems. In this article, we attempt to address this issue by simulating the user interactions with the system as a part of the evaluation process. Particularly, we demonstrate that simulated users find their desired item more efficiently when recommendations are presented as a list of carousels compared to a simple ranked list.

Automated analysis of programming data using code representation methods offers valuable services for programmers, from code completion to clone detection to bug detection. Recent studies show the effectiveness of Abstract Syntax Trees (AST), pre-trained Transformer-based models, and graph-based embeddings in programming code representation. However, pre-trained large language models lack interpretability, while other embedding-based approaches struggle with extracting important information from large ASTs. This study proposes a novel Subtree-based Attention Neural Network (SANN) to address these gaps by integrating different components: an optimized sequential subtree extraction process using Genetic algorithm optimization, a two-way embedding approach, and an attention network. We investigate the effectiveness of SANN by applying it to two different tasks: program correctness prediction and algorithm detection on two educational datasets containing both small and large-scale code snippets written in Java and C, respectively. The experimental results show SANN's competitive performance against baseline models from the literature, including code2vec, ASTNN, TBCNN, CodeBERT, GPT-2, and MVG, regarding accurate predictive power. Finally, a case study is presented to show the interpretability of our model prediction and its application for an important human-centered computing application, student modeling. Our results indicate the effectiveness of the SANN model in capturing important syntactic and semantic information from students' code, allowing the construction of accurate student models, which serve as the foundation for generating adaptive instructional support such as individualized hints and feedback.

The emergence of generative large language models (LLMs) raises the question: what will be its impact on crowdsourcing?Traditionally, crowdsourcing has been used for acquiring solutions to a wide variety of human-intelligence tasks, including ones involving text generation, modification or evaluation.For some of these tasks, models like ChatGPT can potentially substitute human workers.In this study, we investigate whether this is the case for the task of paraphrase generation for intent classification.We apply data collection methodology of an existing crowdsourcing study (similar scale, prompts and seed data) using ChatGPT and Falcon-40B.We show that ChatGPT-created paraphrases are more diverse and lead to at least as robust models.

Recommender systems (RSs) have undoubtedly played a significant role in addressing the information overload problem by efficiently filtering and suggesting relevant items to users. These systems use both explicit and implicit user preferences to filter available data and suggest items that might align with the user’s interests. This can range from recommending movies on a streaming platform based on previous views to suggesting products for purchase based on browsing history. In their early stages, RSs focused on enhancing their algorithmic capabilities to provide accurate recommendations. However, the overemphasis on algorithms resulted in neglecting the human aspect of the user experience. Recognizing this limitation, recent trends in RSs have started to shift their attention toward incorporating Symbiotic Human-Machines Decision Making models. These models aim to provide users with dynamic and persuasive interfaces that empower them to understand and engage better with the recommendations. This shift represents an essential step in creating recommender systems that truly resonate with users and create a more enjoyable, trustable, and user-friendly experience. A crucial aspect of recommender systems’ evolution lies in their proactive nature. Early works focused on designing systems that could proactively anticipate user preferences and needs. While this remains a valuable trait, modern RSs also recognize the importance of giving users control and transparency over their recommendations. Striking the right balance between proactivity and user control ensures that the system supports users without being overly intrusive, thus enhancing their overall satisfaction. These aspects are the main discussion topics of the Joint Workshop on Interfaces and Human Decision Making for Recommender Systems at RecSys’23. In this summary, we introduce the workshop’s motivation and view, review its history, and discuss the most critical issues that deserve attention for future research directions.

To recognize the current and future trends and challenges in computer science education educational materials for the next decade, the authors of this work provide a conversation to voice the computer science community's experience and expertise on these trends. One of the biggest challenges for introductory computing courses in the next few years will be leveraging the new capabilities of Artificial Intelligent systems such as Open AI CodeX and GPT3 that can generate code from a textual description, explain code, and translate code between programming languages. These tools could drastically change how introductory programming is taught by allowing students to focus more on understanding code, modifying code, and testing code than on writing code. Learning content is increasingly shifting from paper textbooks to online learning systems, which include not just traditional text and figures, but increasingly use interactive items to provide students with better explanations and illustrations, extensive practice, and frequent immediate formative feedback, typically at a lower cognitive load than classical programming assignment. We will discuss challenges and opportunities for interoperability with publishing and learning management platforms. Another example is how guided-based instruments, such as peer team learning, open educational resources, or workbooks, are adaptive and hybrid according to students' needs.

The paper focuses on a new type of interactive learning content for SQL programming - worked examples of SQL code. While worked examples are popular in learning programming, their application for learning SQL is limited. Using a novel tool for presenting interactive worked examples, Database Query Analyzer (DBQA), we performed a large-scale randomized controlled study assessing the value of worked examples as a new type of practice content in a database course. We report the results of the classroom study examining the usage and the impact of DBQA. Among other aspects, we explored the effect of textual step explanations provided by DBQA.

The constant increase in the amount of data and information available on the Web has made the development of systems that can support users in making relevant decisions increasingly important. Recommender systems (RSs) have emerged as tools to address this task. RSs use the preferences expressed by a user, either explicitly or implicitly, to filter the available information and proactively suggest items that might be of interest to him or her. Although in early works about the topic there was a strong interest in ways to make such systems proactive, user-friendly, and persuasive, over time they became increasingly focused on the algorithmic component solely. However, this trend is gradually being reversed and always more attention is nowadays placed also on Human Decision Making models that focus on supporting the end user in understanding what is being proposed through RSs by using dynamic and persuasive interfaces. A recommender system should be based on valuable strategies for proactively guiding users to items that match their preferences and therefore should put attention on how it is possible to make this process trustable, pleasant, and user-friendly. Such systems, moreover, should take into account psychological, cognitive and emotional aspects to enable personalization that is appropriate not only to the context of use but also to the psychological reactions of the end user. The workshop provides a venue for works that invest in the design of recommender systems which consider users’ experience during the interaction, as well as for works that explore the implications of human-computer interactions with different theories of human decision-making. In this summary, we introduce the Joint Workshop on Interfaces and Human Decision Making for Recommender Systems at RecSys’22, review its history, and discuss the most important topics considered at the workshop.

Recommending online resources to patients with ovarian cancer and their caregivers is a challenging task. On one hand, the recommended items must be relevant, recent, and reliable. On the other hand, they need to match the user’s levels of disease-specific health literacy. In this demonstration, we describe the overall architecture and key components of HELPeR, a knowledge-adaptive interactive recommender system for ovarian cancer patients and their caregivers.

Adaptive information access systems is one of the most popular types of adaptive systems. It includes adaptive search, recommender system, adaptive navigation support, and adaptive information visualization. This is also one of the oldest types of adaptive systems where the ideas of user control were explored. Motivated by a timely selection of user control as the theme of UMAP 2022, this tutorial provides a systematic review of approaches focused on adding various kinds of user control to adaptive information access systems and discuss lessons learned, prospects, and challenges of this direction of research.

Recommender systems were originally developed as interactive intelligent systems that can proactively guide users to items that match their preferences. Despite its origin on the crossroads of HCI and AI, the majority of research on recommender systems gradually focused on objective accuracy criteria paying less and less attention to how users interact with the system as well as the efficacy of interface designs from users’ perspectives. This trend is reversing with the increased volume of research that looks beyond algorithms, into users’ interactions, decision making processes, and overall experience. The series of workshops on Interfaces and Human Decision Making for Recommender Systems focuses on the ”human side” of recommender systems. The goal of the research stream featured at the workshop is to improve users’ overall experience with recommender systems by integrating different theories of human decision making into the construction of recommender systems and exploring better interfaces for recommender systems. In this summary, we introduce the Joint Workshop on Interfaces and Human Decision Making for Recommender Systems at RecSys’21, review its history, and discuss most important topics considered at the workshop.

We present Grapevine, a user-controlled recommender that enables undergraduate and graduate students to find a suitable research advisor. This system combines the ideas from the areas of exploratory search, user modeling, and recommender systems by employing state-of-the-art knowledge extraction, grape-based recommendation, and an intelligent user interface. In this paper, we demonstrate the system’s key components and how they work as a whole.

Individual differences have been recognized as an important factor in the learning process. However, there are few successes in using known dimensions of individual differences in solving an important problem of predicting student performance and engagement in online learning. At the same time, learning analytics research has demonstrated that the large volume of learning data collected by modern e-learning systems could be used to recognize student behavior patterns and could be used to connect these patterns with measures of student performance. Our paper attempts to bridge these two research directions. By applying a sequence mining approach to a large volume of learner data collected by an online learning system, we build models of student learning behavior. However, instead of following modern work on behavior mining (i.e., using this behavior directly for performance prediction tasks), we attempt to follow traditional work on modeling individual differences in quantifying this behavior on a latent data-driven personality scale. Our research shows that this data-driven model of individual differences performs significantly better than several traditional models of individual differences in predicting important parameters of the learning process, such as success and engagement.

Knowledge Tracing (KT), which aims to model student knowledge level and predict their performance, is one of the most important applications of user modeling. Modern KT approaches model and maintain an up-to-date state of student knowledge over a set of course concepts according to students’ historical performance in attempting the problems. However, KT approaches were designed to model knowledge by observing relatively small problem-solving steps in Intelligent Tutoring Systems. While these approaches were applied successfully to model student knowledge by observing student solutions for simple problems, such as multiple-choice questions, they do not perform well for modeling complex problem solving in students. Most importantly, current models assume that all problem attempts are equally valuable in quantifying current student knowledge. However, for complex problems that involve many concepts at the same time, this assumption is deficient. It results in inaccurate knowledge states and unnecessary fluctuations in estimated student knowledge, especially if students guess the correct answer to a problem that they have not mastered all of its concepts or slip in answering the problem that they have already mastered all of its concepts. In this paper, we argue that not all attempts are equivalently important in discovering students’ knowledge state, and some attempts can be summarized together to better represent student performance. We propose a novel student knowledge tracing approach, Granular RAnk based TEnsor factorization (GRATE), that dynamically selects student attempts that can be aggregated while predicting students’ performance in problems and discovering the concepts presented in them. Our experiments on three real-world datasets demonstrate the improved performance of GRATE, compared to the state-of-the-art baselines, in the task of student performance prediction. Our further analysis shows that attempt aggregation eliminates the unnecessary fluctuations from students’ discovered knowledge states and helps in discovering complex latent concepts in the problems.

Different texts shall by nature correspond to different number of keyphrases.This desideratum is largely missing from existing neural keyphrase generation models.In this study, we address this problem from both modeling and evaluation perspectives.We first propose a recurrent generative model that generates multiple keyphrases as delimiter-separated sequences.Generation diversity is further enhanced with two novel techniques by manipulating decoder hidden states.In contrast to previous approaches, our model is capable of generating diverse keyphrases and controlling number of outputs.We further propose two evaluation metrics tailored towards the variable-number generation.We also introduce a new dataset (ST A C KEX) that expands beyond the only existing genre (i.e., academic writing) in keyphrase generation tasks.With both previous and new evaluation metrics, our model outperforms strong baselines on all datasets.

When we consider our CV, it is full of entities - where we studied, where we worked, who we collaborated with on a project or on a paper. Entities we are linked to are part of our profile and as such they help to understand who we are and what are we interested in. Hence, we adapt the typed entity-relation graph (profile) concept and based on this presentation we propose a personalized multifaceted graph visualization for the entity profile. In the context of an academic conference, we allow scholars to explore a graph of related entities and a word cloud representing the links, providing the user a comprehensive, compact and structured overview about the explored scholar. We evaluated our proposed personalized multifaceted visualization in a user study with encouraging results which showed that this visualization is engaging, easy to use and helpful.

The goal of this workshop is to bring together the existing community of researchers working on Infrastructure Design for Data-Intensive Research in Computer Science Education and a community of Learning at Scale researchers focused on Computer Science Education. While both communities share many similar goals and could greatly benefit from each other work, the interaction between the communities is small. We hope that the proposed workshop will be instrumental in bringing together like-minded researchers from different communities, establishing collaboration, and expanding the scope of infrastructure project to address critical scaling issues.

As an interactive intelligent system, recommender systems are developed to give recommendations that match users’ preferences. Since the emergence of recommender systems, a large majority of research focuses on objective accuracy criteria and less attention has been paid to how users interact with the system and the efficacy of interface designs from users’ perspectives. The field has reached a point where it is ready to look beyond algorithms, into users’ interactions, decision making processes, and overall experience. The series of workshops on Interfaces and Human Decision Making for Recommender Systems focuses on the ”human side” of recommender systems. The goal of the research stream featured at the workshop is to improve users’ overall experience with recommender systems by integrating different theories of human decision making into the construction of recommender systems and exploring better interfaces for recommender systems. In this summary, we introduce 7th Joint Workshop on Interfaces and Human Decision Making for Recommender Systems at RecSys’20, review its history, and discuss most important topics considered at the workshop.

Information-seeking tasks with learning or investigative purposes are usually referred to as exploratory search. Exploratory search unfolds as a dynamic process where the user, amidst navigation, trial and error, and on-the-fly selections, gathers and organizes information (resources). A range of innovative interfaces with increased user control has been developed to support the exploratory search process. In this work, we present our attempt to increase the power of exploratory search interfaces by using ideas of social search—for instance, leveraging information left by past users of information systems. Social search technologies are highly popular today, especially for improving ranking. However, current approaches to social ranking do not allow users to decide to what extent social information should be taken into account for result ranking. This article presents an interface that integrates social search functionality into an exploratory search system in a user-controlled way that is consistent with the nature of exploratory search. The interface incorporates control features that allow the user to (i) express information needs by selecting keywords and (ii) to express preferences for incorporating social wisdom based on tag matching and user similarity. The interface promotes search transparency through color-coded stacked bars and rich tooltips. This work presents the full series of evaluations conducted to, first, assess the value of the social models in contexts independent to the user interface, in terms of objective and perceived accuracy. Then, in a study with the full-fledged system, we investigated system accuracy and subjective aspects with a structural model revealing that when users actively interacted with all of its control features, the hybrid system outperformed a baseline content-based–only tool and users were more satisfied.

The beyond-relevance objectives of recommender systems have been drawing more and more attention. For example, a diversity-enhanced interface has been shown to associate positively with overall levels of user satisfaction. However, little is known about how users adopt diversity-enhanced interfaces to accomplish various real-world tasks. In this article, we present two attempts at creating a visual diversity-enhanced interface that presents recommendations beyond a simple ranked list. Our goal was to design a recommender system interface to help users explore the different relevance prospects of recommended items in parallel and to stress their diversity. Two within-subject user studies in the context of social recommendation at academic conferences were conducted to compare our visual interfaces. Results from our user study show that the visual interfaces significantly reduced the exploration efforts required for given tasks and helped users to perceive the recommendation diversity. We show that the users examined a diverse set of recommended items while experiencing an improvement in overall user satisfaction. Also, the users’ subjective evaluations show significant improvement in many user-centric metrics. Experiences are discussed that shed light on avenues for future interface designs.

Hybrid social recommender systems use social relevance from multiple sources to recommend relevant items or people to users. To make hybrid recommendations more transparent and controllable, several researchers have explored interactive hybrid recommender interfaces, which allow for a user-driven fusion of recommendation sources. In this field of work, the intelligent user interface has been investigated as an approach to increase transparency and improve the user experience. In this paper, we attempt to further promote the transparency of recommendations by augmenting an interactive hybrid recommender interface with several types of explanations. We evaluate user behavior patterns and subjective feedback by a within-subject study (N=33). Results from the evaluation show the effectiveness of the proposed explanation models. The result of post-treatment survey indicates a significant improvement in the perception of explainability, but such improvement comes with a lower degree of perceived controllability.

As an interactive intelligent system, recommender systems are developed to give recommendations that match users' preferences. Since the emergence of recommender systems, a large majority of research focuses on objective accuracy criteria and less attention has been paid to how users interact with the system and the efficacy of interface designs from users' perspectives. The field has reached a point where it is ready to look beyond algorithms, into users' interactions, decision making processes, and overall experience. This workshop will focus on the "human side" of recommender systems research. The workshop goal is to improve users' overall experience with recommender systems by integrating different theories of human decision making into the construction of recommender systems and exploring better interfaces for recommender systems.

Recommender system helps users to reduce information overload. In recent years, enhancing explainability in recommender systems has drawn more and more attention in the field of Human-Computer Interaction (HCI). However, it is not clear whether a user-preferred explanation interface can maintain the same level of performance while the users are exploring or comparing the recommendations. In this paper, we introduced a participatory process of designing explanation interfaces with multiple explanatory goals for three similarity-based recommendation models. We investigate the relations of user perception and performance with two user studies. In the first study (N=15), we conducted card-sorting and semi-interview to identify the user preferred interfaces. In the second study (N=18), we carry out a performance-focused evaluation of six explanation interfaces. The result suggests that the user-preferred interface may not guarantee the same level of performance.

The increasing utilization of massive open online courses has significantly expanded global access to formal education. Despite the technology's promising future, student interaction on MOOCs is still a relatively under-explored and poorly understood topic. This work proposes a multi-level pattern discovery through hierarchical discriminative tensor factorization. We formulate the problem as a hierarchical discriminant subspace learning problem, where the goal is to discover the shared and discriminative patterns with a hierarchical structure. The discovered patterns enable a more effective exploration of the contrasting behaviors of two performance groups. We conduct extensive experiments on several real-world MOOC datasets to demonstrate the effectiveness of our proposed approach. Our study advances the current predictive modeling in MOOCs by providing more interpretable behavioral patterns and linking their relationships with the performance outcome.

This article explores the factors influencing the future citations of conference papers. We concentrated on the explanatory power of early attention on conference papers for citations collected from Google Scholar and Scopus. The early attention data includes users’ online activities in a conference support system: CN3. Bookmarks from the bibliographic management system, Citeulike, were used as a collateral source of early attention. To examine the chronological contributions of 13 factors on citations, a multiple sequential regression analysis was conducted for three timepoints of the publication cycle—paper submission, time of conferences, and months after conferences. Our results illustrate that online readers’ early attention of Citeulike bookmarks were found to have the most influence on the future impact of the conference papers. The early attention records from CN3 made noteworthy improvements to explaining both the Google and Scopus citations as well. We also found that the type of papers the number of papers presented at a conference, and the best article award records were significant factors influencing future citations. However, the magnitude of the effects made by online readers’ early attention from both sources appears to be larger than these three traditional factors.

Information-seeking tasks with learning or investigative purposes are usually referred to as exploratory search. Exploratory search unfolds as a dynamic process where the user, amidst navigation, trial-and-error and on-the-fly selections, gathers and organizes information (resources). A range of innovative interfaces with increased user control have been developed to support exploratory search process. In this work we present our attempt to increase the power of exploratory search interfaces by using ideas of social search, i.e., leveraging information left by past users of information systems. Social search technologies are highly popular nowadays, especially for improving ranking. However, current approaches to social ranking do not allow users to decide to what extent social information should be taken into account for result ranking. This paper presents an interface that integrates social search functionality into an exploratory search system in a user-controlled way that is consistent with the nature of exploratory search. The interface incorporates control features that allow the user to (i) express information needs by selecting keywords and (ii) to express preferences for incorporating social wisdom based on tag matching and user similarity. The interface promotes search transparency through color-coded stacked bars and rich tooltips. In an online study investigating system accuracy and subjective aspects with a structural model we found that, when users actively interacted with all its control features, the hybrid system outperformed a baseline content-based-only tool and users were more satisfied.

The beyond-relevance objectives of recommender systems have been drawing more and more attention. For example, a diversity-enhanced interface has been shown to associate positively with overall levels of user satisfaction. However, little is known about how users adopt diversity-enhanced interfaces to accomplish various real-world tasks. In this paper, we present two attempts at creating a visual diversity-enhanced interface that presents recommendations beyond a simple ranked list. Our goal was to design a recommender system interface to help users explore the different relevance prospects of recommended items in parallel and to stress their diversity. Two within-subject user studies in the context of social recommendation at academic conferences were conducted to compare our visual interfaces. Results from our user study show that the visual interfaces significantly reduced the exploration efforts required for given tasks and helped users to perceive the recommendation diversity. We show that the users examined a diverse set of recommended items while experiencing an improvement in overall user satisfaction. Also, the users» subjective evaluations show significant improvement in many user-centric metrics. Experiences are discussed that shed light on avenues for future interface designs.

As an interactive intelligent system, recommender systems are developed to give recommendations that match users' preferences. Since the emergence of recommender systems, a large majority of research focuses on objective accuracy criteria and less attention has been paid to how users interact with the system and the efficacy of interface designs from users' perspectives. The field has reached a point where it is ready to look beyond algorithms, into users' interactions, decision making processes, and overall experience. his workshop will focus on the "human side" of recommender systems research. The workshop goal is to improve users' overall experience with recommender systems by integrating different theories of human decision making into the construction of recommender systems and exploring better interfaces for recommender systems.

CS Education makes heavy use of online educational tools like IDEs, Learning Management Systems, eTextbooks, interactive programming environments, and other smart content. Instructors and students would benefit from greater interoperability between tools. CS Ed researchers increasingly make use of the large collections of data generated by click streams coming from them. However, we all face barriers that slow progress: (1) Educational tools do not integrate well. (2) Information about CS learning process and outcome data generated by one system is not compatible with that from other systems. (3) CS problem solving and learning (e.g., coding solutions) is different from the type of data (discrete answers to questions or verbal responses) that current educational data mining focuses on. This BOF will discuss ways that we might support and better coordinate efforts to build community and capacity among CS Ed researchers, data scientists, and learning scientists toward reducing these barriers. CS Ed infrastructure should support broader re-use of innovative learning content that is instrumented for rich data collection, formats and tools for analysis of learner data, and best practices to make large collections of learner data available to researchers. Achieving these goals requires engaging a large community of researchers to define, develop, and use critical elements of this infrastructure to address specific data-intensive research questions.

Guiding students to the learning activities that are most appropriate for their current level of knowledge is one of the goals that adaptive educational systems tried to achieve during the last decades. Recently, several attempts have been made to use Open Learner Models (OLM) as a tool for achieving this goal. While the original goal of OLM is to help students reflect about their own learning process, extending OLM with navigation support functionality enables students to take immediate actions towards improving their knowledge. In this work, we attempted to extend the navigation support functionality of OLM by developing a fine-grained OLM that offers student knowledge visualization on both topic and concept levels. The fine-grained OLM enables students to directly explore connections between their knowledge and available learning activities, making an informed decision about their next learning steps. To assess the impact of the new type of OLM, we evaluated several versions of it in a classroom study, while also comparing it with data from our earlier studies that featured a coarse-grained OLM. Our results suggest that the fine-grained OLM considerably impacts student choice of learning activities, making student learning more efficient. We also found that the specific design features of fine-grained OLM could affect students' confidence and persistence while selecting and attempting the learning activities.

The education sector has been undergoing a welcoming change in recent years with the introduction of a wide variety of digital content openly available to students. Due to the volume of this new digital content, it is very difficult for learners to find the needed information at the right time. Digital textbooks, as well-curated domain knowledge sources, could provide a conceptual and physical platform that unites disparate educational resources as one entity. Educational resource linkage, with state-of-the-art techniques, are based on term-level and topic-level representations. However, term-level representations suffer from the term-mismatch problem and often, topics are too broad for linking to other educational resources. To address these challenges, we propose to link educational resources through concept-level representation. The proposed model generates concept embeddings by utilizing domain-specific educational content and external knowledge graph resources to achieve robust and effective concept-level representations. We conducted evaluations of the proposed models on multiple contents linking tasks, and the results demonstrate that concept-level representations perform better than the state-of-the-art representations in helping students to find more learning resources easily. This could increase both students' learning and satisfaction.

Keyphrase provides highly-summative information that can be effectively used for understanding, organizing and retrieving text content.Though previous studies have provided many workable solutions for automated keyphrase extraction, they commonly divided the to-be-summarized content into multiple text chunks, then ranked and selected the most meaningful ones.These approaches could neither identify keyphrases that do not appear in the text, nor capture the real semantic meaning behind the text.We propose a generative model for keyphrase prediction with an encoder-decoder framework, which can effectively overcome the above drawbacks.We name it as deep keyphrase generation since it attempts to capture the deep semantic meaning of the content with a deep learning method.Empirical analysis on six datasets demonstrates that our proposed model not only achieves a significant performance boost on extracting keyphrases that appear in the source text, but also can generate absent keyphrases based on the semantic meaning of the text.Code and dataset are available at https://github.com/memray/seq2seq-keyphrase.

Educational content of today no longer only resides in textbooks and classrooms; more and more learning material is found in a free, accessible form on the Internet. Our long-standing vision is to transform this web of educational content into an adaptive, web-scale "textbook", that can guide its readers to most relevant "pages" according to their learning goal and current knowledge. In this paper, we address one core, long-standing problem towards this goal: identifying outcome and prerequisite concepts within a piece of educational content (e.g., a tutorial). Specifically, we propose a novel approach that leverages textbooks as a source of distant supervision, but learns a model that can generalize to arbitrary documents (such as those on the web). As such, our model can take advantage of any existing textbook, without requiring expert annotation. At the task of predicting outcome and prerequisite concepts, we demonstrate improvements over a number of baselines on six textbooks, especially in the regime of little to no ground-truth labels available. Finally, we demonstrate the utility of a model learned using our approach at the task of identifying prerequisite documents for adaptive content recommendation --- an important step towards our vision of the "web as a textbook".

As intelligent interactive systems, recommender systems focus on determining predictions that fit the wishes and needs of users. Still, a large majority of recommender systems research focuses on accuracy criteria and much less attention is paid to how users interact with the system, and in which way the user interface has an influence on the selection behavior of the users. Consequently, it is important to look beyond algorithms. The main goals of the IntRS workshop are to analyze the impact of user interfaces and interaction design, and to explore human interaction with recommender systems from a human decision making perspective. Methodologies for evaluating these aspects are also within the scope of the workshop.

Open Learner Models (OLM) show the learner model to users to assist their self-regulated learning by, for example, helping prompt reflection, facilitating planning and supporting navigation. OLMs can show different levels of detail of the underlying learner model, and can also structure the information differently. As a result, a trade-off may exist between the potential for better support for learning and the complexity of the information shown. This paper investigates students' perceptions about whether offering more and richer information in an OLM will result in more effective support for their self-regulated learning. In a first study, questionnaire responses relating to designs for six visualisations of varying complexity led to the implementation of three variations on one of the designs. A second controlled study involved students interacting with these variations. The study revealed that the most useful variation for searching for suitable learning material was a visualisation combining a basic coloured grid, an extended bar chart-like visualisation indicating related concepts, and a learning gauge.

Stereotypes are frequently used in real life to classify students according to their performance in class. In literature, we can find many references to weaker students, fast learners, struggling students, etc. Given the lack of detailed data about students, these or other kinds of stereotypes could be potentially used for user modeling and personalization in the educational context. Recent research in MOOC context demonstrated that data-driven learner stereotypes could work well for detecting and preventing student dropouts. In this paper, we are exploring the application of stereotype-based modeling to a more challenging task -- predicting student problem-solving and learning in two programming courses and two MOOCs. We explore traditional stereotypes based on readily available factors like gender or education level as well as some advanced data-driven approaches to group students based on their problem-solving behavior. Each of the approaches to form student stereotype cohorts is validated by comparing models of student learning: do students in different groups learn differently? In the search for the stereotypes that could be used for adaptation, the paper examines ten approaches. We compare the performance of these approaches and draw conclusions for future research.

Complex skill mastery requires not only acquiring individual basic component skills, but also practicing integrating such basic skills. However, traditional approaches to knowledge modeling, such as Bayesian knowledge tracing, only trace knowledge of each decomposed basic component skill. This risks early assertion of mastery or ineffective remediation failing to address skill integration. We introduce a novel integration-level approach to model learners' knowledge and provide fine-grained diagnosis: a Bayesian network based on a new kind of knowledge graph with progressive integration skills. We assess the value of such a model from multifaceted aspects: performance prediction, parameter plausibility, expected instructional effectiveness, and real-world recommendation helpfulness. Our experiments based on a Java programming tutor show that proposed model significantly improves two popular multiple-skill knowledge tracing models on all these four aspects.

A social recommender system aims to provide useful suggestion to the user and prevent social overload problem. Most of the research efforts are spent on push high relevant item on top of the ranked list, using a weight ensemble approach. However, we argue the ``learned'' static fusion is not enough to specific contexts. In this paper, we develop a series visual recommendation components and control panel for the user to interact with the recommendation result of an academic conference. The system offers a better recommendation transparency and user-driven fusion through recommended sources. The experiment result shows the user did fuse the different recommended sources and exploration patterns among tasks. The post-study survey is positively associated with the system and explanation function effectiveness. This finding shed light on the future research of design a recommender system with human intervention and the interface beyond the static ranked list.

Following the popularity of Wikipedia, community authoring systems are increasingly in use as content sharing outlets. As such, a Web-based portal for sharing of user-generated content (e.g., course notes, quiz answers, etc.) shows prospect to be a great tool for social E-Learning. Among others, students are expected to be active contributors in such systems in order to offer and receive peer-help. However, privacy and territoriality concerns can be potential barriers to wide adoption of such technology. Understanding the preference for sharing learning content is the first step to address privacy and territoriality concerns of content providers. The authors conduct a survey among students in four university courses in order to learn their preference for sharing notes and quiz answers with three target groups: instructor, peer, and stranger (i.e., someone outside their class). The authors also examine the preference for acceptable method of sharing by inquiring about three methods: “anonymous sharing,” “pseudonymous sharing,” and “sharing with name”. They further investigate the importance of “content type,” “sharing method,” and “accessor type” on the preference for sharing. The survey also reveals respondents' self-reported reasons for controlling access to their generated learning content. The survey data indicate that even though the respondents have various levels of concerns, almost all of them are willing to share. The authors observe relationships between content type and respondents' preference over each of these parameters: accessor type, commentator type, and sharing method.

We present the Mastery Grids system, an intelligent interface for online learning content that combines open learner modeling (OLM) and social comparison features. We grounded the design of Mastery Grids in self-regulated learning and learning motivation theories, as well as in our past work in social comparison, OLM, and adaptive navigation support. The force behind the interface is the combination of adaptive navigation functionality with the mastery-oriented aspects of OLM and the performance-oriented aspects of social comparison. We examined different configurations of Mastery Grids in two classroom studies and report the results of analysis of log data and survey responses. The results show how Mastery Grids interacts with different factors, like gender and achievement-goal orientation, and ultimately, its impact on student engagement, performance, and motivation.

As intelligent interactive systems, recommender systems focus on determining predictions that fit the wishes and needs of users. Still, a large majority of recommender systems research focuses on accuracy criteria and much less attention is paid to how users interact with the system, and in which way the user interface has an influence on the selection behavior of the users. Consequently, it is important to look beyond algorithms. The main goals of the IntRS workshop are to analyze the impact of user interfaces and interaction design, and to explore human interaction with recommender systems. Methodologies for evaluating these aspects are also within the scope of the workshop.

Code examples are commonly used learning resources that help students grasp various programming structures and concepts. However, example code usually requires explanations about what each line or part of the code does. Otherwise, students may find it difficult to follow an example. In this paper, we compare two types of code examples that use different techniques to describe important concepts in the code: annotated and animated examples. The former displays an explanation for a subset of lines in plain text, whereas the latter visualizes code execution. We studied the use and impact of these enhanced examples, provided as non-mandatory practice content, in three introductory Java courses. Our results suggest that animated examples are more engaging and have a positive impact on students' learning. As compared to annotated examples, students spent more time with animated examples and more likely completed them. Also, a positive relationship was found between the number of explored animated examples and the overall course grade.

Various e-learning systems that provide electronic textbooks are gathering data on large numbers of student reading interactions. This data can potentially be used to model student knowledge acquisition. However, reading activity is often overlooked in canonical student modeling. Prior studies modeling learning from reading either estimate student knowledge at the end of all reading activities, or use quiz performance data with expert-crafted knowledge components (KCs). In this work, we demonstrate that the dynamic modeling of student knowledge is feasible and that automatic text analysis can be applied to save expert effort. We propose a data-driven approach for dynamic student modeling in textbook-based learning. We formulate the problem of modeling learning from reading as a reading-time prediction problem, reconstruct existing popular student models (such as Knowledge Tracing) and explore two automatic text analysis approaches (bag-of-words-based and latent semantic-based) to build the KC model. We evaluate the proposed framework using a dataset collected from a Human-Computer Interaction course. Results show that our approach for reading modeling is plausible; the proposed Knowledge Tracing-based student model reliably outperforms baselines and the latent semantic-based approach can be a promising way to construct a KC model. Serving as the first step to model dynamic knowledge in textbook-based learning, our framework can be applied to a broader context of open-corpus personalized learning.

Although the volume of online educational resources has dramatically increased in recent years, many of these resources are isolated and distributed in diverse websites and databases. This hinders the discovery and overall usage of online educational resources. By using linking between related subsections of online textbooks as a testbed, this paper explores multiple knowledge-based content linking algorithms for connecting online educational resources. We focus on examining semantic-based methods for identifying important knowledge components in textbooks and their usefulness in linking book subsections. To overcome the data sparsity in representing textbook content, we evaluated the utility of external corpuses, such as more textbooks or other online educational resources in the same domain. Our results show that semantic modeling can be integrated with a term-based approach for additional performance improvement, and that using extra textbooks significantly benefits semantic modeling. Similar results are obtained when we applied the same approach to other domains.

Several approaches have been researched to help people deal with abundance of information. An important feature pioneered by social tagging systems and later used in other kinds of social systems is the ability to explore different community relevance prospects by examining items bookmarked by a specific user or items associated by various users with a specific tag . A ranked list of recommended items offered by a specific recommender engine can be considered as another relevance prospect. The problem that we address is that existing personalized social systems do not allow their users to explore and combine multiple relevance prospects. Only one prospect can be explored at any given time—a list of recommended items, a list of items bookmarked by a specific user, or a list of items marked with a specific tag. In this article, we explore the notion of combining multiple relevance prospects as a way to increase effectiveness and trust. We used a visual approach to recommend articles at a conference by explicitly presenting multiple dimensions of relevance. Suggestions offered by different recommendation techniques were embodied as recommender agents to put them on the same ground as users and tags. The results of two user studies performed at academic conferences allowed us to obtain interesting insights to enhance user interfaces of personalized social systems. More specifically, effectiveness and probability of item selection increase when users are able to explore and interrelate prospects of items relevance—that is, items bookmarked by users, recommendations and tags. Nevertheless, a less-technical audience may require guidance to understand the rationale of such intersections.

Open user modeling has been perceived as an important mechanism to enhance the effectiveness of personalization. However, several studies have reported that open and editable user models can harm the performance of personalized search systems. This paper re-examines the value of open and editable user models in the context of personalized search. We implemented a personalized search system with 2D user manipulatable visualization and concept-based user model components. A user study result suggests that the proposed visualization-based open user modeling approach can be beneficial for adaptive search.

As the heterogeneity of data sources are increasing on the web, and due to the sparsity of data in each of these data sources, cross-domain recommendation is becoming an emerging research topic in the recent years. Cross-domain collaborative filtering aims to transfer the user rating pattern from source (auxiliary) domains to a target domain for the purpose of alleviating the sparsity problem and providing better target recommendations. However, the studies so far have either focused on a limited number of domains that are assumed to be related to each other (such as books and movies), or a division of the same dataset (such as movies) into different domains based on an item characteristic (such as genre). In this paper, we study a broad set of domains and their characteristics to understand the factors that affect the success or failure of cross-domain collaborative filtering, the amount of improvement in cross-domain approaches, and the selection of best source domains for a specific target domain. We propose to use Canonical Correlation Analysis (CCA) as a significant major factor in finding the most promising source domains for a target domain, and suggest a cross-domain collaborative filtering based on CCA (CD-CCA) that proves to be successful in using the shared information between domains in the target recommendations.

No abstract available.

Crowdsourcing has emerged as a way to harvest social wisdom from thousands of volunteers to perform a series of tasks online. However, little research has been devoted to exploring the impact of various factors such as the content of a resource or crowdsourcing interface design on user tagging behavior. Although images' titles and descriptions are frequently available in image digital libraries, it is not clear whether they should be displayed to crowdworkers engaged in tagging. This paper focuses on offering insight to the curators of digital image libraries who face this dilemma by examining (i) how descriptions influence the user in his/her tagging behavior and (ii) how this relates to the (a) nature of the tags, (b) the emergent folksonomy, and (c) the findability of the images in the tagging system. We compared two different methods for collecting image tags from Amazon's Mechanical Turk's crowdworkers—with and without image descriptions. Several properties of generated tags were examined from different perspectives: diversity, specificity, reusability, quality, similarity, descriptiveness, and so on. In addition, the study was carried out to examine the impact of image description on supporting users' information seeking with a tag cloud interface. The results showed that the properties of tags are affected by the crowdsourcing approach. Tags from the “with description” condition are more diverse and more specific than tags from the “without description” condition, while the latter has a higher tag reuse rate. A user study also revealed that different tag sets provided different support for search. Tags produced “with description” shortened the path to the target results, whereas tags produced without description increased user success in the search task.

Many pieces of educational software are underused by students. Open learning model and social visualization are two approaches which have been helpful in ameliorating that low usage problem. This article introduces a fusion of these two ideas in a form of social progress visualization. A classroom evaluation indicates that this combination may be effective in engaging students, guiding them to suitable content, and enabling faster content access.

Research in recommender systems has traditionally focused on improving the predictive accuracy of recommendations by developing new algorithms or by incorporating new sources of data. However, several studies have shown that accuracy does not always correlate with a better user experience, leading to recent research that puts emphasis on Human-Computer Interaction in order to investigate aspects of the interface and user characteristics that influence the user experience on recommender systems. Following this new research this paper presents SetFusion, a visual user-controllable interface for hybrid recommender system. Our approach enables users to manually fuse and control the importance of recommender strategies and to inspect the fusion results using an interactive Venn diagram visualization. We analyze the results of two field studies in the context of a conference talk recommendation system, performed to investigate the effect of user controllability in a hybrid recommender. Behavioral analysis and subjective evaluation indicate that the proposed controllable interface had a positive effect on the user experience.

As an interactive intelligent system, recommender systems are developed to give predictions that match users preferences. Since the emergence of recommender systems, a large majority of research focuses on objective accuracy criteria and less attention has been paid to how users interact with the system and the efficacy of interface designs from the end-user perspective. The field has reached a point where it is ready to look beyond algorithms, into users interactions, decision making processes and overall experience. Accordingly, the goals of this workshop (Int[email protected]) are to explore the human aspects of recommender systems, with a particular focus on the impact of interfaces and interaction design on decision-making and user experiences with recommender systems, and to explore methodologies to evaluate these human aspects of the recommendation process that go beyond traditional automated approaches.

This paper investigates recommendation algorithms to suggest talks of interest to attendees of research conferences. In this study, based on a social conference support system Conference Navigator 3 (CN3), we explored three kinds of knowledge sources to generate recommendations: users' preference about talks (CN3 bookmarks), users' social networks (research collaboration network and CN3 following network) and talk content information (titles and abstracts). Using these sources, we explored a diverse set of algorithms from non-personalized community vote-based recommendations and conventional collaborative filtering recommendations to hybrid recommendations such as social network-based (SN) recommendations boosted by content information of talks. We found that SN recommendations fused with content information outperformed the other approaches. Moreover, for cold-start users who have an insufficient number of bookmarks to express their preferences, the recommendations based on their social connections also generated significantly better suggestions than the other approaches. Between two kinds of social networks that we considered as foundations of recommendations, there was no significant difference in the quality of the recommendations.

We present a theoretical foundation and empirical findings demonstrating the effectiveness of personality-targeted design. Much like a medical treatment applied to a person based on his specific genetic profile, we argue that theory-driven, personality-targeted UI design can be more effective than design applied to the entire population. The empirical exploration focused on two settings, two populations and two personality traits: Study 1 shows that users' extraversion level moderates the relationship between the UI cue of audience size and users' contribution. Study 2 demonstrates that the effectiveness of social anchors in encouraging online contributions depends on users' level of emotional stability. Taken together, the findings demonstrate the potential and robustness of the interactionist approach to UI design. The findings contribute to the HCI community, and in particular to designers of social systems, by providing guidelines to targeted design that can increase online participation.

This paper presents our attempt to develop a personalized exam preparation tool for Java/OOP classes based on a fine-grained concept model of Java knowledge. Our goal was to explore two most popular student model-based approaches: open student modeling and problem sequencing. The result of our work is a Java exam preparation tool, Knowledge Zoom. The tool combines an open concept-level student model component, Knowledge Explorer and a concept-based sequencing component, Knowledge Maximizer into a single interface. This paper presents both components of Knowledge Zoom, reports results of its evaluation, and discusses lessons learned.

Research on recommender systems has traditionally focused on the development of algorithms to improve accuracy of recommendations. So far, little research has been done to enable user interaction with such systems as a basis to support exploration and control by end users. In this paper, we present our research on the use of information visualization techniques to interact with recommender systems. We investigated how information visualization can improve user understanding of the typically black-box rationale behind recommendations in order to increase their perceived relevance and meaning and to support exploration and user involvement in the recommendation process. Our study has been performed using TalkExplorer, an interactive visualization tool developed for attendees of academic conferences. The results of user studies performed at two conferences allowed us to obtain interesting insights to enhance user interfaces that integrate recommendation technology. More specifically, effectiveness and probability of item selection both increase when users are able to explore and interrelate multiple entities -- i.e. items bookmarked by users, recommendations and tags.

When evaluating a network topology, occasionally data structures cannot be segmented into absolute, heterogeneous groups. There may be a spectrum to the dataset that does not allow for this hard clustering approach and may need to segment using fuzzy/overlapping communities or cliques. Even to this degree, when group members can belong to multiple cliques, there leaves an ever present layer of doubt, noise, and outliers caused by the overlapping clustering algorithms. These imperfections can either be corrected by an expert user to enhance the clustering algorithm or to preserve their own mental models of the communities. Presented is a visualization that models overlapping community membership and provides an interactive interface to facilitate a quick and efficient means of both sorting through large network topologies and preserving the user's mental model of the structure.

Social learning has confirmed its value in enhancing the learning outcomes across a wide spectrum. To support social learning, a visual approach is a common technique to represent and organize multiple students' data in an informative way. This paper presents a design of comparative social visualization for E-learning, which encourages information discovery and social comparisons. Classroom studies confirmed the motivational impact of personalized social guidance provided by the visualization in the target context. The visualization encouraged students to do some work ahead of the course schedule. Moreover, class leaders provided an implicit social guidance for the rest of the class and successfully led the way to discover the most relevant resources creating good trails for the rest of the class. We summarized the evidence of students' engagement and performance through the social visualization interface.

BooksOnline'12, the fifth workshop in the series, aims to offer a forum for bringing together expertise from academia, industry and libraries to facilitate the exchange of research results and technology in the field of digital libraries with specific focus on online books and complementary social media. The focus of this year's workshop is "engaging reading experiences", starting from the act of deciding what to read, through the exploration and interpretation of a book's content, to sharing the overall experience. Within this overall umbrella theme, the accepted papers naturally showed three salient themes: (1) Search and Discovery, (2) Personalization and Recommendation, and Reading Experiences beyond Text. The contributions demonstrate a range of technologies, including a collaborative tabletop visual approach to support the searching and discovery of books, co-citation methods to enhance document retrieval; exploring open issues in audio-book production to support non-text based reading and improving e-book accessibility; new approaches to recommendation that take into account writing style as well as looking specifically to young readers and their needs in order to develop recommendation tools that consider both content and reading level and match these against the readers' specific interests and reading ability. Following in the theme of the reader playing a central role in the future of our digital era, we are honored to welcome Maribeth Back from FX Palo Alto and Natasa Milic-Frayling from Microsoft Research as our keynote speakers.

This paper investigates various recommendation algorithms to recommend relevant talks to attendees of research conferences. We explored three sources of information to generate recommendations: users’ preference about items (i.e. talks), users’ social network and content of items. In order to find o

Students usually do not study individually; instead, they tend to study collaboratively. The “power of known peers” can be embraced to provide implicit navigation support for collaborative social e-learning environments. In this paper, we present a novel approach to collaborative e-learning through

Successful contributors in online communities go through a lifecycle of membership status starting from the periphery and moving to the core over time. We argue that a personalized incremental engagement strategy can mobilize a larger proportion of members through this process. This paper presents our approach to identify a progression of contributions in an online community and the design of personalized incremental engagement strategy which will drive users' contributions one step forward. The results of our field study show that the personalized incremental approach significantly boosts contributions in an online community for academic conferences.

Over the last 10 years two major research directions explored the benefits of visualizing student learning progress. One stream of research on learning performance visualization attempts to build a visual presentation of students' learning progress, targeting the needs of instructors and academic advisors. The other stream of research on Open Student Modeling (OSM) attempts to visualize the state of individual student's knowledge and present the visualization directly to the student. The results of the studies in that area show that, presenting students with basic representation of their knowledge will result in facilitating their metacognitive activities and promoting self-reflection and awareness. This paper tries to study the impact of a more sophisticated form of performance visualization on students. We believe that our visualization tool can positively influence students by granting them the opportunity to get a view of their performance in the content of the class progress. Moreover, we tried to boost their motivation by building a positive sense of competition using a representation of average class performance. In this paper we present study comparing two groups of students, one using the visualization and another without visualization. The results of the study shows that: 1) the students are likely to use the social visualization tool during the whole semester to monitor their progress in comparison with their peers; 2) the visualization tool encourages students to use the learning materials in a more continuous manner during the whole semester and 3) students will achieve a higher success rate in answering self-assessment quizzes.

The BooksOnline Workshop series aims to foster the discussion and exchange of research ideas towards addressing challenges and exploring opportunities around large collections of digital books and complementary media. The fourth workshop in the series, BooksOnline'11 pays special attention to the role of social media and the phenomena of crowdsourcing in the context of online books, which is expected to be key in defining new user experiences in digital libraries and on the Web. The workshop boasts a high quality program, including keynote addresses by Ville Miettinnen, CEO of Microtask and Adam Farquhar, Head of Digital Library Technology at The British Library. From the accepted papers two main themes became salient: 1) Information retrieval and information extraction methods focused on enhancing digital libraries, and 2) Studies and analyses of reading experience and behaviour. This paper provides an overview of the workshop and the accepted contributions.

Reading is one of the most important skills in today's society. The ubiquity of this activity has naturally affected many information systems; the only goal of some is the presentation of textual information. One concrete task often performed on a computer and involving reading is finding relevant parts of text. In the current study, we investigated if word-level relevance, defined as a binary measure of an individual word being congruent with the reader's current informational needs, could be inferred given only the text and eye movements of readers. We found that the number of fixations, first-pass fixations, and the total viewing time can be used to predict the relevance of sentence-terminal words. In light of what is known about eye movements of readers, knowing which sentence-terminal words are relevant can help in an unobtrusive identification of relevant sentences.

No abstract available.

This paper presents Progressor, a visualization of open student models intended to increase the student's motivation to progress on educational content. The system visualizes not only the user's own model, but also the peers' models. It allows sorting the peers' models using a number of criteria, including the overall progress and the progress on a specific topic. Also, in this paper we present results of a classroom study confirming our hypothesis that by showing a student the peers' models and ranking them by progress it is possible to increase the student's motivation to compete and progress in e-learning systems.

The goal of the 3rd BooksOnline Workshop is to bring together researchers and industry practitioners in information retrieval, digital libraries, e-books, human computer interaction, publishing industry, and online book services to foster progress on addressing challenges and exploring opportunities around large collections of digital books and complementary media. Towards this goal, the workshop programme consists of contributions both from academia and industry, including two keynote talks: James Crawford from Google Books and John Mark Ockerbloom from the University of Pennsylvania.

Social navigation and social tagging technologies enable user communities to assemble the collective wisdom, and use it to help community members in finding the right information. However, it takes a significantly-sized community to make a social system truly useful. The question addressed in this p

No abstract available.

This paper aims to combine information about users' self-defined social connections with traditional collaborative filtering (CF) to improve recommendation quality. Specifically, in the following, the users' social connections in consideration were groups. Unlike other studies which utilized groups inferred by data mining technologies, we used the information about the groups in which each user explicitly participated. The group activities are centered on common interests. People join a group to share and acquire information about a topic as a form of community of interest or practice. The information of this group activity may be a good source of information for the members. We tested whether adding the information from the users' own groups or group members to the traditional CF-based recommendations can improve the recommendation quality or not. The information about groups was combined with CF using a mixed hybridization strategy. We evaluated our approach in two ways, using the Citeulike data set and a real user study.

Social tagging systems pose new challenges to developers of recommender systems. As observed by recent research, traditional implementations of classic recommender approaches, such as collaborative filtering, are not working well in this new context. To address these challenges, a number of research groups worldwide work on adapting these approaches to the specific nature of social tagging systems. In joining this stream of research, we have developed and evaluated two enhancements of user-based collaborative filtering algorithms to provide recommendations of articles on Cite ULike, a social tagging service for scientific articles. The result obtained after two phases of evaluation suggests that both enhancements are beneficial. Incorporating the number of raters into the algorithms, as we do in our NwCF approach, leads to an improvement of precision, while tag-based BM25 similarity measure, an alternative to Pearson correlation for calculating the similarity between users and their neighbors, increases the coverage of the recommendation process.

Information retrieval is one of the most popular information access methods for overcoming the information overload problem of the Web. However, its interaction model is still utilizing the old text-based ranked lists and static interaction algorithm. In this paper, we introduce our adaptive visualization approach for searching the Web, which we call Adaptive VIBE. It is an extended version of a reference point-based spatial visualization algorithm, and is designed to serve as a user interaction module for a personalized search system. Personalized search can incorporate dynamic user interests and different contexts, improving search results. When it is combined with adaptive visualization, it can encourage users to become involved in the search process more actively by exploring the information space and learning new facts for effective searching. In this paper, we introduce the rationale and functions of our adaptive visualization approach and discuss the approaches' potential to create a better search environment for the Web.

Rich, interactive eLearning tools receive a lot of attention nowadays from both practitioners and researchers. However, broader dissemination of these tools is hindered by the technical difficulties of their integration into existing platforms. This article explores the technical and conceptual problems of using several interactive educational tools in the context of a single course. It presents an integrated Exploratorium for database courses, an experimental platform, which provides personalized access to several types of interactive learning activities. Several classroom studies of the Exploratorium have demonstrated its value in both the integration of several tools and the provision of personalized access.

Problem-tracing questions are popular among teachers of various programming languages. In an assessment mode these questions allows to evaluate student knowledge of language semantics. In a self-assessment mode, they provide an excellent learning tool. A 2004 ITiCSE working group report [4] stressed the importance of this type of questions to build foundation of higher-level knowledge. Yet the use of problem-tracing questions is still limited due to a large authoring overhead. To resolve this bottleneck, we explored the idea of parameterized question generation [2]. We developed QuizPACK [1], a system which can generate parameterized problem-tracing questions for C programming language. We also developed QuizGuide [1], a personalized guidance system for QuizPACK, which models student knowledge and guides students individually to most appropriate questions to try. The results of our studies demonstrated that QuizPACK strongly benefits student knowledge and that QuizGuide personalized guidance technology increased student ability to answer questions correctly and encouraged them to use the system more extensively (which, in turn, positively impacted their knowledge) [1]. However, parameterized questions in area of C programming were not as diverse from the complexity point of view as parameterized questions explored in other areas such as physics [2]. As a result, it was left unclear whether personalized guidance technology can successfully guide students to a broader range of questions from relatively simple to very difficult.The work reported in this poster expands our work on parameterized questions to a more sophisticated domain of object-oriented Java programming, which allowed us to introduce questions of much broader. Capitalizing on our experience with QuizPACK, we developed QuizJET (Java Evaluation Toolkit), which supports authoring, delivery, and evaluation of parameterized questions for Java [3]. We also implemented JavaGuide system (Figure 1), which provides personalized guidance for QuizJET questions. We assessed the impact of adaptive navigation support to student work with questions of different complexity as well as the impact of this technology on weaker and stronger students. The results of two classroom studies indicate that personalized guidance encouraged students to use parameterized questions more extensively and also helped them to access right questions at the right time. Students were 2.5 times more likely to answer a quiz correctly with personalized guidance than without such it. In addition, we found that personalized guidance especially benefited weak students to achieve scores comparable with the scores of strong students on each complexity level of questions.

Collaborative tagging systems pose new challenges to the developers of recommender systems. As observed by recent research, traditional implementations of classic recommender approaches, such as collaborative filtering, are not working well in this new context. To address these challenges, a number of research groups worldwide work on adapting these approaches to the specific nature of collaborative tagging systems. In joining this stream of research, we have developed and compared three variants of user-based collaborative filtering algorithms to provide recommendations of articles on CiteULike. The first approach, Classic Collaborative filtering (CCF) uses Pearson correlation to calculate similarity between users and a classic adjusted ratings formula to rank the recommendations. The second approach, Neighbor-weighted Collaborative Filtering, takes into account the number of raters in the ranking formula of the recommendations. The third approach explores an innovative way to form the user neighborhood based on a modified version of the Okapi BM25 model over users' tags. Our results suggest that both alterations of CCF are beneficial. Incorporating the number of raters into the algorithms leads to an improvement of precision, while tag-based BM25 can be considered as an alternative to Pearson correlation to calculate the similarity between users and their neighbors.

In this paper, we present an open architecture that combines different SQL learning tools in an integrated Exploratorium for database courses. The integrated Exploratorium provides a unique learning environment that allows database students to take complimentary advantages of multiple advanced learning tools.

Personalized Web search has emerged as one of the hottest topics for both the Web industry and academic researchers. However, the majority of studies on personalized search focused on a rather simple type of search, which leaves an important research topic - the personalization in exploratory searches - as an under-studied area. In this paper, we present a study of personalization in task-based information exploration using a system called TaskSieve. TaskSieve is a Web search system that utilizes a relevance feedback based profile, called a "task model", for personalization. Its innovations include flexible and user controlled integration of queries and task models, task-infused text snippet generation, and on-screen visualization of task models. Through an empirical study using human subjects conducting task-based exploration searches, we demonstrate that TaskSieve pushes significantly more relevant documents to the top of search result lists as compared to a traditional search system. TaskSieve helps users select significantly more accurate information for their tasks, allows the users to do so with higher productivity, and is viewed more favorably by subjects under several usability related characteristics.

This paper reviews our work on providing students interactive access to annotated program examples. We review our experience with WebEx, the system that allows students to explore examples line by line. After that we present NavEx, an adaptive environment for accessing interactive programming examples. NavEx enhances WebEx with a specific kind of adaptive navigation support known as adaptive annotation. The classroom study of NavEx discovered that adaptive navigation support can visibly increase student motivation to work with nonmandatory educational content. NavEx boosted the overall amount of work done and the average length of a session. In addition, various features of NavEx were highly regarded by the students.

The goal of this paper is to detail the integration of two "social Web" technologies - social search and social navigation - and to highlight the benefits of such integration on two levels. Firstly, both technologies harvest and harness "community wisdom" and in an integrated system each of the search and navigation components can benefit from the additional community wisdom gathered by the other when assisting users to locate relevant information. Secondly, by integrating search and browsing we facilitate the development of a unique interface that effectively blends search and browsing functionality as part of a seamless social information access service. This service allows users to effectively combine their search and browsing behaviors. In this paper we will argue that this integration provides significantly more than the simple sum of the parts.

Adaptive information filtering is a promising tool for both casual Web news readers and professional intelligence analysts. Adaptive filtering augments the traditional query- or profile-based rankings provided by search engines. An interesting research challenge in this context is to offer users more control over the rankings by letting them mediate between the two extremes - query- and profile-based rankings. To address this challenge, we developed an adaptive relevance-based visual exploration tool based on the VIBE (visual information browsing environment) visualization approach, which was previously developed at our School. This paper presents the rationale and functionality of this visual exploration tool and reports the results of its preliminary evaluation.

In profile-based or content-based adaptive systems, one of the open research questions is how frequently the user's profile and the list of recommended items should be updated. Different systems tend to choose one of the two extremes. Some systems do it once per session (thus called between-session update strategy), whereas some others update whenever there is feedback (called instant update strategy). This paper presents our attempt to assess the value of keeping the list of recommended items up-to-date in the context of task-based information exploration. We conducted controlled studies involving human users performing realistic tasks using two systems that have the same adaptive filtering engine but with the above two different update strategies. Our results show that the between-session strategy helped to find better quality information, and received better subjects' responses about its usefulness and usability. However, it prolonged the selection of useful passages, whereas the instant update strategy helped subjects to obtain almost all of their selected passages (>98%) within the first 5 minutes. Based on the results, we hypothesize that the best strategy for updating might be a hybrid between the two update strategies, where both adaptability and stability can be achieved.

Over the last five years, a range of projects have focused on progressively more elaborated techniques for adaptive news delivery. However, the adaptation process in these systems has become more complicated and thus less transparent to the users. In this paper, we concentrate on the application of open user models in adding transparency and controllability to adaptive news systems. We present a personalized news system, YourNews, which allows users to view and edit their interest profiles, and report a user study on the system. Our results confirm that users prefer transparency and control in their systems, and generate more trust to such systems. However, similar to previous studies, our study demonstrate that this ability to edit user profiles may also harm the system.s performance and has to be used with caution.

Adaptive explanatory visualization is an attempt to integrate two promising approaches to program visualization: adaptive visualization and explanatory visualization. The goal of this paper is to demonstrate the ideas of adaptive explanatory visualization using a practical example. The paper introduces the WADEIn II system for the visualization of expression evaluation in the C programming language, shows how expression evaluation visualizations can be made adaptive, and explains our approach to the adaptive generation of explanations.

A number of research teams are working to organize personalized access to the modern repositories of educational resources. The goal of personalized access is to help students locate resources that match their individual goals, interests, and current knowledge. The project presented in this paper is focused on the least explored way of personalized access - adaptive visualization. Here, we present the NavEx ADVISE visualization system, which provides personalized access to a repository of educational examples. The system combines spatial, similarity-based visualization with adaptive annotations of resources. The spatial layout and the adaptive annotations are generated using a knowledge-based indexing of examples with domain concepts.

We explored two approaches for encouraging introductory programming students to use the web-based, self-assessment system, QuizPACK. An "organizational" approach applied specially constructed classroom quizzes, while the "technical" approach introduced adaptive guidance. Our studies demonstrated that each of these caused a dramatic increase in system use. This approach could be useful in many other contexts, when an educationally beneficial system is underused by students.

article Share on Preface to the special issue on automated assessment of programming assignments Authors: Peter Brusilovsky University of Pittsburgh, PA University of Pittsburgh, PAView Profile , Colin Higgins University of Nottingham, UK University of Nottingham, UKView Profile Authors Info & Claims Journal on Educational Resources in ComputingVolume 5Issue 3September 2005 pp 1–eshttps://doi.org/10.1145/1163405.1163406Published:01 September 2005Publication History 22citation811DownloadsMetricsTotal Citations22Total Downloads811Last 12 Months7Last 6 weeks1 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my Alerts New Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteGet Access

Individualized exercises are a promising feature in promoting modern e-learning. The focus of this article is on the QuizPACK system, which is able to generate parameterized exercises for the C language and automatically evaluate the correctness of student answers. We introduce QuizPACK and present the results of its comprehensive classroom evaluation during four consecutive semesters. Our studies demonstrate that when QuizPACK is used for out-of-class self-assessment, it is an exceptional learning tool. The students' work with QuizPACK significantly improved their knowledge of semantics and positively affected higher-level knowledge and skills. The students themselves praised the system highly as a learning tool. We also demonstrated that the use of the system in self-assessment mode can be significantly increased by basing later classroom paper-and-pencil quizzes on QuizPACK questions, motivating students to practice them more.

Content creation for a language learning system is a time-consuming, laborious, and expensive task. An important part of content in a language learning system is the illustrative content, namely example sentences. We discuss a solution to reuse illustrative content in a Web-based language learning system for the German and Italian languages. Our approach explores fully the potential of hypertext, which allows building complex networks of small pieces of learning material. A first evaluation shows promising results.

This paper discusses a need to provide adaptive navigation support for students accessing large numbers of interactive examples in Web-enhanced education. We introduce the system NavEx that is able to provide adaptive annotation of programming examples without any need of manual indexing. NavEx uses innovative algorithms for prerequisite/outcome indexing of learning material and an efficient knowledge-based approach for adaptive annotation.

Electronic Performance Support Systems (EPSS) is a challenging application area for developing intelligent interfaces. Some possible scenarios for using domain, task, and user models for adaptive performance support were explored in the context of the Adaptive Diagnostics and Personalized Technical Support (ADAPTS) project. ADAPTS provides an intelligent, adaptive EPSS for maintaining complex equipment.

Adaptive hypermedia is a new direction of research within the area of adaptive and user model-based interfaces. Adaptive hypermedia (AH) systems build a model of the individual user and apply it for adaptation to that user, for example, to adapt the content of a hypermedia page to the user's knowledge and goals, or to suggest the most relevant links to follow. AH systems are used now in several application areas where the hyperspace is reasonably large and where a hypermedia application is expected to be used by individuals with different goals, knowledge and backgrounds. This paper is a review of existing work on adaptive hypermedia. The paper is centered around a set of identified methods and techniques of AH. It introduces several dimensions of classification of AH systems, methods and techniques and describes the most important of them.