Assessing handwritten mathematical solutions is essential for identifying students’ weaknesses and fostering personalized learning. However, scaling such assessment remains challenging for Learning Analytics, which has traditionally focused on digital or typed data. The current study investigated the potential of Large Language Models (LLMs) to automating the assessment of handwritten mathematical solutions and explore how they can be incorporated into large scale learning analytics pipelines. We curated 300 student solution images, annotated them using to a taxonomy of math error types, and compared open-source (Qwen2.5-7B and Gemma3 12B-IT) and closed-source (Gemini 2.0 Flash and GPT-4) LLMs. Two prompting strategies were tested: from adapted from related work and one tailed to the taxonomy of math error types using established prompt design principles. The results revealed that LLMs, particularly Gemini, achieved strong to moderate performance in diagnosing and classifying student errors, while exposing recurring model specific errors. These findings highlight both the promise and limitations of LLMs for integrating handwritten work in LA and recommend that learning analytics practitioners and researchers combine careful model selection, principled prompt design, and error-level analysis to develop AI-powered LA systems that are accurate, equitable, and pedagogically actionable.

International audience

Contains fulltext : 317127.pdf (Publisher’s version ) (Open Access)

Assessing short answers in educational settings is challenging due to the need for scalability and accuracy, which led to the field of Automatic Short Answer Grading (ASAG). Traditional machine learning models, such as ensemble and embeddings, have been widely researched in ASAG, but they often suffer from generalizability issues. Recently, Large Language Models (LLMs) emerged as an alternative to optimize ASAG systems. However, previous research has failed to present a comprehensive analysis of LLMs' performance powered by prompt engineering strategies and compare its capabilities to traditional models. This study presents a comparative analysis between traditional machine learning models and GPT-4 in the context of ASAG. We investigated the effectiveness of different models and text representation techniques and explored prompt engineering strategies for LLMs. The results indicate that traditional machine learning models outperform LLMs. However, GPT-4 showed promising capabilities, especially when configured with optimized prompt components, such as few-shot examples and clear instructions. This study contributes to the literature by providing a detailed evaluation of LLM performance compared to traditional machine learning models in a multilingual ASAG context, offering insights for developing more efficient automatic grading systems.

Question-answering systems facilitate adaptive learning and respond to student queries, making education more responsive. Despite that, challenges such as natural language understanding and context management complicate their widespread adoption, where Large Language Models (LLMs) offer a promising solution. However, existing research is predominantly focused on English, proprietary models, and often limited to a single question type, subject, or skill, leaving a gap in understanding LLMs' performance in languages like Brazilian Portuguese and across questions of various characteristics. This study investigates how LLMs could be integrated in an educational question-answering system efficiently to answer different question types (multiple-choice, cloze, open-ended), subjects (mathematics and Portuguese language), and skills (summation/subtraction, multiplication, interpretation, and grammar), evaluating answers by GPT-4 - the main LLM at the time of writing - and Sabiá - the open-source Brazilian Portuguese LLM - based on grades assigned by two experienced teachers. Overall, both LLMs demonstrated strong overall performance, with mean scores close to 9.8 out of 10. However, specific challenges emerged, with distinct strengths and weaknesses observed for each model, such as GPT-4's error in a multiple-choice subtraction question and Sabiá's misinterpretation of a cloze question.

To meet the growing demand for future professionals who can present information to an audience and create quality written products, educators are increasingly assigning writing assignments that require students to gather information from multiple sources, reorganise and reinterpret knowledge from source materials, and plan for rhetorical structure goals in order to meet the task requirements. When evaluating an essay coherence, scorers manually look for the presence of required rhetorical categories, which takes time. Supervised Machine Learning (ML) techniques have proven to be an effective tool for automatic detection of rhetorical categories that approximate students’ cognitive engagement with source information. Previous studies that addressed this problem used relatively small datasets and reported relatively low kappa scores for accuracy, limiting the use of such models in real-world scenarios. Moreover, to empower educators to effectively evaluate the overall quality of students’ writing, the associations between the sequential patterns of rhetorical categories in students’ writing and writing performance must be examined, which remains largely unexplored in educational domain. Therefore, to fill these gaps, our study aimed to i) investigate the impact of data augmentation approaches on the performance of deep learning algorithms in classifying rhetorical categories in student essays according to Bloom‘s taxonomy ii) and explore the sequential patterns of rhetorical categories in students’ writing that can influence writing performance. Our findings showed that deep learning-based model BERT on Easy Data Augmentation (EDA) based augmented data achieved 20% higher Cohen’s kappa than normal (non-augmented) data, and we discovered that students in different performance groups were statistically different in terms of rhetorical patterns. Our proposed study is valuable in terms of building a data analytic foundation that can be used to create formative feedback on students’ writings based on the patterns of rhetorical categories to improve essay quality.

The workshop will give educators an introduction to graphical data analysis techniques for exploring, summarizing, and effectively communicating data using Visual Blocks (Blockly) and Jupiter Notebooks. Participants will gain skills to create and interpret various graphs and plots, fostering insights into data patterns and relationships. The emphasis will be on adeptly selecting visualizations. Upon completion of this course, educators should be proficient in: Understanding the principles of graphical data analysis and their practical applications; and creating and interpreting diverse graph types and plots. The workshop will further delve into statistical methods essential for data analysis, guiding educators on employing descriptive statistics to explore data and derive meaningful business insights. The focus will be on fostering an understanding of statistical concepts and their practical application, as well as the effective interpretation and communication of findings. Additionally, the program will introduce educators to machine learning regressors, with a primary focus on linear regression. Participants will learn the skills to train, evaluate, and apply regressors to predict continuous target variables from input features. The emphasis will be on grasping and applying machine learning principles, as well as effectively interpreting and communicating model predictions. Furthermore, participants will engage in hands-on experience with machine learning classifiers, encompassing logistic regression, decision trees, naive Bayes, and neural networks. Educators will acquire the expertise to train, evaluate, and apply classifiers for predicting categorical target variables from input features. The emphasis here is on understanding and applying machine learning principles, as well as adeptly interpreting model predictions.

Abstract Peer feedback has been widely used in computer-supported collaborative learning (CSCL) setting to improve students’ engagement with massive courses. Although the peer feedback process increases students’ self-regulatory practice, metacognition, and academic achievement, instructors need to go through large amounts of feedback text data which is much more time-consuming. To address this challenge, the present study proposes an automated content analysis approach to identify relevant categories in peer feedback based on traditional and sequence-based classifiers using TF-IDF and content-independent features. We use a data set from an extensive course (N = 231 students) in the setting of engineering higher education. In particular, a total of 2,444 peer feedback messages were analyzed. The CRF classification model based on the TF-IDF features achieved the best performance. The results illustrate that the ability to scale up the automatic analysis of peer feedback provides new opportunities for student-improved learning and improved teacher support in higher education at scale.

Texts produced by the Brazilian judiciary have a complex and technical vocabulary, with elaborate use of the Portuguese language and many legal terms difficult to be understood, generating a barrier in communication between the judiciary and the population. In this sense, the Automatic Text Simplification (ATS), activity of the Natural Language Processing (NLP) area, can be applied to improve the readability of these types of text using specialized algorithms, and promote scalability in simplifying them, in view of the great demand in the courts. In this context, this article presents an evaluation of four methods of state of the art in text simplification, evaluated according to readability metrics, to improve the quality of existing texts in the judicial summaries, dataset containing 100 summaries of the Federal Regional Court of the 5th Region (TRF5) and another 100 of the Federal Supreme Court (STF). The methods MUSS(EN), MUSS(PT), Transformers and NMT + Attention were tested, and the results of the simplifications exceeded the FRE readability index of the original texts, making them more readable.

Interest in data science has been growing across industries - both STEM and non-STEM. Non-STEM students often have difficulties with programming and data analysis tools. These entry barriers can be minimized, and these concepts can be easily absorbed when using visual tools. Thus, for this specific audience, the use of visual tools has been essential for teaching data science. Several of these tools are available, but they all have limitations. This work presents Blockly-DS: a new tool capable of assisting in teaching data science to a non-STEM audience. The Blockly-DS tool is being tested in two Brazilian higher education institutions, one, IBMEC, a business undergraduate university, and the other, FIAP, a STEM school that offers an MBA as well as corporate and undergraduate courses. The preliminary results presented in this article refers to a validation with two groups of training sessions for junior financial analysts of a major Brazilian bank in partnership with FIAP.

Essay writing has become one of the most common learning tasks assigned to students enrolled in various courses at different educational levels, owing to the growing demand for future professionals to effectively communicate information to an audience and develop a written product (i.e. essay). Evaluating a written product requires scorers who manually examine the existence of rhetorical categories, which is a time-consuming task. Machine Learning (ML) approaches have the potential to alleviate this challenge. As a result, several attempts have been made in the literature to automate the identification of rhetorical categories using Rhetorical Structure Theory (RST). However, RST do not provide information regarding students’ cognitive level, which motivates the use of Bloom’s Taxonomy. Therefore, in this research we propose to: i) investigate the extent to which classification of rhetorical categories can be automated based on Bloom’s taxonomy by comparing the traditional ML classifiers with the pre-trained language model BERT, ii) explore the associations between rhetorical categories and writing performance. Our results showed that BERT model outperformed the traditional ML-based classifiers with 18% better accuracy, indicating it can be used in future analytics tool. Moreover, we found a statistical difference between the associations of rhetorical categories in low-achiever, medium-achiever and high-achiever groups which implies that rhetorical categories can be predictive of writing performance.

Feedback is an effective way to assist students in achieving learning goals. The conceptualisation of feedback is gradually moving from feedback as information to feedback as a learner-centred process. To demonstrate feedback effectiveness, feedback as a learner-centred process should be designed to provide quality feedback content and promote student learning outcomes on the subsequent task. However, it remains unclear how instructors adopt the learner-centred feedback framework for feedback provision in the teaching practice. Thus, our study made use of a comprehensive learner-centred feedback framework to analyse feedback content and identify the characteristics of feedback content among student groups with different performance changes. Specifically, we collected the instructors’ feedback on two consecutive assignments offered by an introductory to data science course at the postgraduate level. On the basis of the first assignment, we used the status of student grade changes (i.e., students whose performance increased and those whose performance did not increase on the second assignment) as the proxy of the student learning outcomes. Then, we engineered and extracted features from the feedback content on the first assignment using a learner-centred feedback framework and further examined the differences of these features between different groups of student learning outcomes. Lastly, we used the features to predict student learning outcomes by using widely-used machine learning models and provided the interpretation of predicted results by using the SHapley Additive exPlanations (SHAP) framework. We found that 1) most features from the feedback content presented significant differences between the groups of student learning outcomes, 2) the gradient boost tree model could effectively predict student learning outcomes, and 3) SHAP could transparently interpret the feature importance on predictions.

Textual cohesion is an essential aspect of a formally written text, related to linguistic mechanisms that connect elements such as words, sentences, and paragraphs. Several studies have proposed approaches to estimate textual cohesion in essays automatically. There is limited research that aims to study the extent to which the use of machine learning approaches can predict the textual cohesion of essays written in different languages (not just English). This paper reports on the findings of a study that aimed to propose and evaluate approaches that automatically estimate the cohesion of essays in Portuguese and English. The study proposed regression-based models grounded in conventional feature-based machine learning methods and deep learning-based pre-trained language models. The study also examined the explainability of automated approaches to scrutinize their predictions. We analyzed two datasets composed of 4,570 (Portuguese) and 7,101 (English) essays. The results demonstrate that a deep learning-based model achieved the best performance on both datasets with a moderate Pearson correlation with human-rated cohesion scores. However, the explainability of the automatic cohesion estimations based on conventional machine learning models offered a stronger potential than that of the deep learning model.

The sequencing of sampling algorithms has shown to be a promising approach in generating balanced versions of unbalanced data. Sequencing allows different algorithms of under-sampling and/or over-sampling to be performed in sequence, producing a resulting balanced database. However, defining the most appropriate sequence of sampling algorithms is challenging. This article treats the sequencing problem as a combinatorial optimization task and proposes a multi-objective optimization method to seek promising solutions that maximize the performance of classifiers both in accuracy and in F1-score. The results showed that the proposed method was capable of finding optimized sequences that improved the performance of the classifiers, obtaining statistically better results, mainly in F1- score, when compared with competing methods, in most of the selected unbalanced problems.

Stakeholder engagement is a key aspect for the successful implementation of Learning Analytics (LA) in Higher Education Institutions (HEIs). Studies in Europe and Latin America (LATAM) indicate that, overall, instructors and students have positive views on LA adoption, but there are differences between their ideal expectations and what they consider realistic in the context of their institutions. So far, very little has been found about stakeholders’ views on LA in Brazilian higher education. By replicating the survey conducted in other countries, in seven Brazilian HEIs, we found convergences both with Europe and LATAM, reinforcing the need for local diagnosis and indicating the risk of assuming a ”LATAM identity”. Our findings contribute to building a corpus of knowledge on stakeholders expectations with a contextualised comprehension of the gaps between ideal and predicted scenarios, which can inform institutional policies for LA implementation in Brazil.

Roles that learners assume during online discussions are an important aspect of educational experience. The roles can be assigned to learners and/or can spontaneously emerge through student-student interaction. While existing research proposed several approaches for analytics of emerging roles, there is limited research in analytic methods that can i) automatically detect emerging roles that can be interpreted in terms of higher-order constructs of collaboration; ii) analyse the extent to which students complied to scripted roles and how emerging roles compare to scripted ones; and iii) track progression of roles in social knowledge progression over time. To address these gaps in the literature, this paper propose a network-analytic approach that combines techniques of cluster analysis and epistemic network analysis. The method was validated in an empirical study discovered emerging roles that were found meaningful in terms of social and cognitive dimensions of the well-known model of communities of inquiry. The study also revealed similarities and differences between emerging and script roles played by learners and identified different progression trajectories in social knowledge construction between emerging and scripted roles. The proposed analytic approach and the study results have implications that can inform teaching practice and development techniques for collaboration analytics.

This research aimed to explore the relationship between different indicators of the depth and quality of participation in computer-mediated learning environments. By using network analyses and statistical tests, we discovered significant associations between the cognitive presence phases of the Community of Inquiry framework and speech acts, and examined the impact of two different instructional interventions on these associations. We found that there are strong associations between some speech acts and cognitive presence phases. In addition, the study revealed that the association between speech acts and cognitive presence is moderated by external facilitation, but not affected by user role assignment. The results suggest that speech acts can plausibly be used to provide feedback in relation to cognitive presence and can potentially be used to increase the generalizability of cognitive presence classification.

Brazilian universities have included essay writing assignments in the entrance examination procedure to select prospective students. The essay scorers manually look for the presence of required Rhetorical Structure Theory (RST) categories and evaluate essay coherence. However, identifying RST categories is a time-consuming task. The literature reported several attempts to automate the identification of RST categories in essays with machine learning. Still, previous studies have focused on using machine learning algorithms trained on content-dependent features that can diminish classification performance, leading to over-fitting and hindering model generalisability. Therefore, this paper proposes: (i) the analysis of state-of-the-art classifiers and content-independent features to the task of RST rhetorical moves; (ii) a new approach that considers the sequence of the text to extract features – i.e. sequential content-independent features; (iii) an empirical study about the generalisability of the machine learning models and sequential content-independent features for this context; (iv) the identification of the most predictive features for automated identification of RST categories in essays written in Portuguese. The best performing classifier, XGBoost, based on sequential content-independent features, outperformed the classifiers used in the literature and are based on traditional content-dependent features. The XGBoost classifier based on sequential content-independent features also reached promising accuracy when tested for generalisability.

Deep Convolutional Neural Networks (CNNs) have reached the attention in the last decade due to their successful application to many computer vision domains. Several handcrafted architectures have been proposed in the literature, with increasing depth and millions of parameters. However, the optimal architecture size and parameters setup are dataset-dependent and challenging to find. For addressing this problem, this work proposes a Multi-Objective Grammatical Evolution framework to automatically generate suitable CNN architectures (layers and parameters) for a given classification problem. For this, a Context-free Grammar is developed, representing the search space of possible CNN architectures. The proposed method seeks to find suitable network architectures considering two objectives: accuracy and F1-score. We evaluated our method on CIFAR-10, and the results obtained show that our method generates simpler CNN architectures and overcomes the results achieved by larger (more complex) state-of-the-art CNN approaches and other grammars.

Supervised machine learning models have been widely used to address the classification of messages in online discussions. Supervised learning algorithms require a large set of annotated data to accurately create a predictive model. However, data annotation is a complex task due to three factors: (i) depends on specialists to accurately label data; (ii) it is often a time-consuming and labour-intensive work,and(iii) in educational settings, it is not always easy to collect a substantial volume of data required by the machine learning algorithms. This paper presents an active learning-based approach that can reduce the amount of annotated data required to build machine learning models for the classification of educational data. The results obtained show that with only 20% of the annotated data, the proposed approach achieved similar results to those presented in the previous works that used the complete databases to train the machine learning model.

According to the World Health Organization, an estimated one billion people live with a disability. Millions of them are non-verbal and also experience motor-skill challenges. The restrictions on participation and communication caused by such disabilities often lead to discrimination and social exclusion, including the lack of access to formal education. Augmentative and Alternative Communication (AAC) is a method to afford communication for people with speech impairment. Software applications that implement AAC bring benefits of adaptability and personalization over traditional paper-based methods, but their usability needs improvement, particularly to increase user autonomy. This paper presents an interface redesign of the Livox AAC application, and a new user onboarding process based on user research to adjust the interface to user needs, contributing to user autonomy on AAC use.

This paper reports the findings of a study that measured the effectiveness of employing automatic text translation methods in automated classification of online discussion messages according to the categories of social and cognitive presences. Specifically, we examined the classification of 1,500 Portuguese and 1,747 English discussion messages using classifiers trained on the datasets before and after the application of text translation. While the English model generated, with the original and translated texts, achieved results (accuracy and Cohen’s κ) similar to those of the previously reported studies, the translation to Portuguese led to a decrease in the performance. The indicates the general viability of the proposed approach when converting the text to English. Moreover, this study highlighted the importance of different features and resources, and the limitations of the resources for Portuguese as reasons of the results obtained.

In online learning, teachers need constant feedback about their students’ progress and regulation needs. Learning Analytics Dashboards for process-oriented feedback can be a valuable tool for this purpose. However, few such dashboards have been proposed in literature, and most of them lack empirical validation or grounding in learning theories. We present a teacher-facing dashboard for process-oriented feedback in online learning, co-designed and evaluated through an iterative design process involving teachers and visualization experts. We also reflect on our design process by discussing the challenges, pitfalls, and successful strategies for building this type of dashboard.

Feedback plays a crucial role in student learning. Learning analytics (LA) has demonstrated potential in addressing prominent challenges with feedback practice, such as enabling timely feedback based on insights obtained from large data sets. However, there is insufficient research looking into relations between student expectations of feedback and their experience with LA-based feedback. This paper presents a pilot study that examined students’ experience of LA-based feedback, offered with the OnTask system, taking into consideration the factors of students’self-efficacy and self-regulation skills. Two surveys were carried out at a Brazilian university, and the results highlighted important implications for LA-based feedback practice, including leveraging the ‘partnership’ between the human teacher and the computer, and developing feedback literacy among learners.

The DocEng'2020 Competition on Extractive Text Summarization assessed the performance of six new methods and fourteen classical algorithms for extractive text sumarization. The systems were evaluated using the CNN-Corpus, the largest test set available today for single document extractive summarization using two different strategies and the ROUGE and the direct match measures.

The unprecedented growth of textual content on the Web made essential the development of automatic or semi-automatic techniques to help people to find valuable information in such a huge heap of text data. Automatic text summarization is one of such techniques that is being pointed out as offering a viable solution in such a chaotic scenario. Extractive text summarization, in particular, selects a set of sentences from a text according to specific criteria. Strategies for extractive summarization can benefit from preprocessing techniques that emphasize the relevance or infor-mativeness of sentences with respect to the selection criteria. This paper tests such a hypothesis using sentence simplification methods. Four methods are used to simplify a corpus of news articles in English: a rule-based method, an optimization method, a supervised deep learning model and an unsupervised deep learning model. The simplified outputs are summarized using 14 sentence selection strategies. The combinations of simplification and summarization methods are compared with the baseline --- the summarized corpus without previous simplification --- with a quantitative analysis, which suggests sentence compression with restrictions and models learned from large parallel corpora tend to perform better and yield gains over summarization without prior simplification.

Learning Analytics is a new field in education whose adoption can bring benefits for teaching and learning processes. However, many higher education institutions may not be ready to start using learning analytics due to challenges such as organizational culture, infrastructure, and privacy. In this context, Maturity Models (MMs) can support institutions to systematize their processes, enabling them to progress successively in the learning analytics adoption. MMs are used in different fields to support the improvement of processes, describing them in terms of maturity levels, and identifying enhancements that could lead an organization to higher levels of such maturity. Thus, this paper presents an outline of a MM for Learning Analytics adoption in higher education institutions, describing its levels and areas, together with its development methodology.

This paper presents a study that examined automated cross-language classification of online discussion messages for the levels of cognitive presence, a key construct from the widely used Community of Inquiry (CoI) model of online learning. Specifically, we examined the classification of 1,500 Portuguese language discussion messages using a classifier trained on a corpus of the 1,747 English language discussion messages. In the study, a random forest classifier was developed using a small set of 108 validated indicators of psychological processes, linguistic coherence, and online discussion structure. The classifier obtained 67% accuracy and Cohen's κ of 0.32, showing a moderate level of inter-rater agreement above chance and the general viability of the proposed approach. Most importantly, the findings suggest that certain aspects of cognitive presence construct are highly generalizable and transfer across different languages. Finally, the paper also presents a novel method for addressing class imbalance problem using a generic algorithm heuristic technique, which provided substantial improvements over the use of imbalanced dataset. Results and practical implications are further discussed.

Feedback is a crucial element in helping students identify gaps and assess their learning progress. In online courses, feedback becomes even more critical as it is one of the resources where the teacher interacts directly with the student. However, with the growing number of students enrolled in online learning, it becomes a challenge for instructors to provide good quality feedback that helps the student self-regulate. In this context, this paper proposed a content analysis of feedback text provided by instructors based on different indicators of good feedback. A random forest classifier was trained and evaluated at different feedback levels. The results achieved outcomes up to 87% and 0.39 of accuracy and Cohen's κ, respectively. The paper also provides insights into the most influential textual features of feedback that predict feedback quality.

Learning Analytics and Knowledge (LAK) and Educational Data Mining (EDM) are two of the most popular venues for researchers and practitioners to report and disseminate discoveries in data-intensive research on technology-enhanced education. After the development of about a decade, it is time to scrutinize and compare these two venues. By doing this, we expected to inform relevant stakeholders of a better understanding of the past development of LAK and EDM and provide suggestions for their future development. Specifically, we conducted an extensive comparison analysis between LAK and EDM from four perspectives, including (i) the topics investigated; (ii) community development; (iii) community diversity; and (iv) research impact. Furthermore, we applied one of the most widely-used language modeling techniques (Word2Vec) to capture words used frequently by researchers to describe future works that can be pursued by building upon suggestions made in the published papers to shed light on potential directions for future research.

Several tools to support learning processes based on educational data have emerged from research on Learning Analytics (LA) in the last few years. These tools aim to support students and instructors in daily activities, and academic managers in making institutional decisions. Although the adoption of LA tools is spreading, the field still needs to deepen the understanding of the contexts where learning takes place, and of the views of the stakeholders involved in implementing and using these tools. In this sense, the SHEILA framework proposes a set of instruments to perform a detailed analysis of the expectations and needs of different stakeholders in higher education institutions, regarding the adoption of LA. Moreover, there is a lacuna in research on stakeholders' expectations from LA outside the Global North. Therefore, this paper reports on the findings of the application of interviews and focus groups, based on the SHEILA framework, with students and teaching staff from a Brazilian public university, to investigate their perceptions of the potential benefits and risks of using LA in higher education in the country. Findings indicate that there is a high interest in using LA for improving the learning experience, in particular, being able to provide personalized feedback, to adapt teaching practices to students' needs, and to make evidence-based pedagogical decisions. From the analysis of these perspectives, we point to opportunities for using LA in Brazilian higher education.

This paper presents an approach to automatic labeling of the content of messages in online discussion according to the categories of social presence. To achieve this goal, the proposed approach is based on a combination of traditional text mining features and word counts extracted with the use of established linguistic frameworks (i.e., LIWC and Coh-metrix). The best performing classifier obtained 0.95 and 0.88 for accuracy and Cohen's kappa, respectively. This paper also provides some theoretical insights into the nature of social presence by looking at the classification features that were most relevant for distinguishing between the different categories. Finally, this study adopted epistemic network analysis to investigate the structural construct validity of the automatic classification approach. Namely, the analysis showed that the epistemic networks produced based on messages manually and automatically coded produced nearly identical results. This finding thus produced evidence of the structural validity of the automatic approach.

The DocEng'19 Competition on Extractive Text Summarization assessed the performance of two new and fourteen previously published extractive text sumarization methods. The competitors were evaluated using the CNN-Corpus, the largest test set available today for single document extractive summarization.

This paper details the features and the methodology adopted in the construction of the CNN-corpus, a test corpus for single document extractive text summarization of news articles. The current version of the CNN-corpus encompasses 3,000 texts in English, and each of them has an abstractive and an extractive summary. The corpus allows quantitative and qualitative assessments of extractive summarization strategies.

This paper details the development and features of the CNN-corpus in Spanish, possibly the largest test corpus for single document extractive text summarization in the Spanish language. Its current version encompasses 1,117 well-written texts in Spanish, each of them has an abstractive and an extractive summary. The development methodology adopted allows good-quality qualitative and quantitative assessments of summarization strategies for tools developed in the Spanish language.

Feedback is an essential component of any learning experience. It allows students to identify gaps in their learning and improve their self-regulation. However, providing useful feedback is a challenging and time-consuming task. In digital learning environments, this challenge is even more significant due to a large number of students. Thus, this paper reports on the findings of an analysis of the quality of feedback provided by instructors in an online course. The paper also proposes a supervised machine learning algorithm that can identify the presence of good practices in feedback messages sent to students in a digital learning environment. The results reveal the most commonly used kinds of feedback and how to identify them automatically. The results of the study could potentially be used to improve the quality of the feedback provided by instructors in online education.

In recent years, several tools to support learning processes have emerged from the research on Learning Analytics (LA). However, little attention has been given to the contexts where learning takes place, and to the stakeholders involved in implementing and using these tools. Following the SHEILA Framework, we present the results of interviews and focus groups with students from a Brazilian public university on their perceptions of the potential and risks of using LA in higher education. Findings indicate great difficulties to use the Learning Management System; lack of effective communication with teaching staff; lack of quality and timely feedback; high perceived value of personalization; rejection of competition and rankings; issues with self-confidence; and trustworthiness of the educational institution. From the analysis of students' perspectives, we point to opportunities to use LA in the context of a Brazilian public university.

Student retention and learning in STEM disciplines is a growing problem. The 2012 report by the US President's Council of Advisors on Science and Technology (PCAST) predicts a future deficit in science, engineering, and mathematics (STEM) in the following decade and emphasizes the importance of addressing this issue. With this as a motivating factor, the OSBLE+ Social Programming Environment (SPE) was used to leverage social and programming data for the basis of automatically generated prompts inserted into the SPE. These prompts were designed to stimulate help-seeking, help-giving, and social interaction in the learning environment. A social network analysis was performed in order to determine whether exposure to the automated interventions would positively affect the relationship among students over time. Results of this study suggest that students in the experimental treatment who were presented with automated prompts developed more connected and social networks than those in the control treatment.

This paper proposes a topic model-based approach to extract students' weaknesses and strength based on Latent Dirichlet Allocation (LDA). Our approach combines textual data extracted from online discussion forums written by students with external sources like Wikipedia. The results show the effectiveness of the proposed approach to create a user profile based on the topics covered by the students in discussion forums.

This paper presents an approach to studying relationships between students' social presence and course topics from transcripts of asynchronous discussions in online learning environments. Specifically, the paper uses topic modeling and epistemic network analysis to investigate how students' social presence is expressed across different course topics. Finally, we show how this method can be adopted to examine how students' social presence changed due to an instructional intervention. The results of this study and its implications are further discussed.

Mobile devices such as smart phones and tablets are omnipresent in modern societies. Such devices allow browsing the Internet. This paper briefly describes two tools for news article summarization in mobile devices that attempts to automatically collect and sieve the most important information of news article in WebPages.

The existing automatic text summarization systems whenever applied to web-pages of news articles show poor performance as the text is encapsulated within a HTML page. This paper takes advantage of the link identification and content extraction techniques. The results show the validity of such a strategy.

Text summarization is the process of automatically creating a shorter version of one or more text documents. This paper presents a qualitative and quantitative assessment of the 22 state-of-the-art extractive summarization systems using the CNN corpus, a dataset of 3,000 news articles.

An efficient way to automatically classify documents may be provided by automatic text summarization, the task of creating a shorter text from one or several documents. This paper presents an assessment of the 15 most widely used methods for automatic text summarization from the text classification perspective. A naive Bayes classifier was used showing that some of the methods tested are better suited for such a task.

The need for automatic generation of summaries gained importance with the unprecedented volume of information available in the Internet. Automatic systems based on extractive summarization techniques select the most significant sentences of one or more texts to generate a summary. This article makes use of Machine Learning techniques to assess the quality of the twenty most referenced strategies used in extractive summarization, integrating them in a tool. Quantitative and qualitative aspects were considered in such assessment demonstrating the validity of the proposed scheme. The experiments were performed on the CNN-corpus, possibly the largest and most suitable test corpus today for benchmarking extractive summarization strategies.

Text summarization is the process of creating a shorter version of one or more text documents. Automatic text summarization has become an important way of finding relevant information in large text libraries or in the Internet. Extractive text summarization techniques select entire sentences from documents according to some criteria to form a summary. Sentence scoring is the technique most used for extractive text summarization, today. Depending on the context, however, some techniques may yield better results than some others. This paper advocates the thesis that the quality of the summary obtained with combinations of sentence scoring methods depend on text subject. Such hypothesis is evaluated using three different contexts: news, blogs and articles. The results obtained show the validity of the hypothesis formulated and point at which techniques are more effective in each of those contexts studied.

The text data available on the Internet is not only huge in volume, but also in diversity of subject, quality and idiom. Such factors make it infeasible to efficiently scavenge useful information from it. Automatic text summarization is a possible solution for efficiently addressing such a problem, because it aims to sieve the relevant information in documents by creating shorter versions of the text. However, most of the techniques and tools available for automatic text summarization are designed only for the English language, which is a severe restriction. There are multilingual platforms that support, at most, 2 languages. This paper proposes a language independent summarization platform that provides corpus acquisition, language classification, translation and text summarization for 25 different languages.

Sentence similarity is used to measure the degree of likelihood between sentences. It is used in many natural language applications, such as text summarization, information retrieval, text categorization, and machine translation. The current methods for assessing sentence similarity represent sentences as vectors of bag of words or the syntactic information of the words in the sentence. The degree of likelihood between phrases is calculated by composing the similarity between the words in the sentences. Two important concerns in the area, the meaning problem and the word order, are not handled, however. This paper proposes a new sentence similarity assessment measure that largely improves and refines a recently published method that takes into account the lexical, syntactic and semantic components of sentences. The new method proposed here was benchmarked using a publically available standard dataset. The results obtained show that the new similarity assessment measure proposed outperforms the state of the art systems and achieve results comparable to the evaluation made by humans.

Relation extraction (RE) aims at finding the way entities, such as person, location, organization, date, etc., depend upon each other in a text document. Ontology Population, Automatic Summarization, and Question Answering are fields in which relation extraction offers valuable solutions. A relation extraction method based on inductive logic programming that induces extraction rules suitable to identify semantic relations between entities was proposed by the authors in a previous work. This paper proposes a method to simplify graph-based representations of sentences that replaces dependency graphs of sentences by simpler ones, keeping the target entities in it. The goal is to speed up the learning phase in a RE framework, by applying several rules for graph simplification that constrain the hypothesis space for generating extraction rules. Moreover, the direct impact on the extraction performance results is also investigated. The proposed techniques outperformed some other state-of-the-art systems when assessed on two standard datasets for relation extraction in the biomedical domain.

Text summarization is the process of automatically creating a shorter version of one or more text documents. In this context, word-based, sentence-based and graph-based methods approaches are largely used. Among these, graph based methods for automatic text summarization produce summaries based on the relationships between sentences. These relationships may also support the creation of several text processing applications such as extractive and abstractive summaries, question-answering and information retrieval systems, among others. A new graph model for text processing applications is proposed in this paper. It relies on four dimensions (similarity, semantic similarity, co reference, discourse information) to create the graph. The rationale behind the proposal presented here is resorting to more dimensions than previous works, and taking into account co reference resolution, taking into account to the role of pronouns in connecting the sentences. Co reference was not used in any previous graph based summarization technique. An experiment was performed using the Text Rank algorithm with the presented approach, on the CNN corpus. The results show that the model proposed here outperforms the current approaches both quantitatively and qualitatively.