This work-in-progress research paper analyzes student activity as captured by the sequence of programming submissions they make. Although students who complete assignments in a computing course are often assessed in a summative way through the submission of a completed program, their activities during development provide additional insights into their work and problem-solving processes that cannot otherwise be captured. In the context of a data structures & algorithms course taught at the sophomore level, we first examined the number of attempts that students made while completing assignments throughout the semester. We then examined the cumulative portion of the class that had submitted an assignment relative to the due date, and the time periods after assignment release that students are active. Our work helps to illustrate how a programming activity trace provides a unique source of information on how a submission evolves. It was found that even with only feedback as motivation, students made many submissions and that number increased during the semester. Our initial work indicates that a substantial amount of data is available and helps suggest what aspects might provide information useful for further analytics using traditional statistics or machine-learning approaches.

Students taking computing courses develop skills by applying programming to various problems. In the past decade, courses have started to move from manually graded assessments to those which can be automated. A typical motivation for the use of automated assessment is to enable scaling, which is of particular importance for courses taught with high enrollment. However, automated assessment provides additional advantages such as reproducibility and rapid feedback. We have developed an automated assessment platform that performs a combination of static and dynamic analysis to evaluate student work. Our focus has been not on scaling but rather on serving student educational outcomes, both at the student level (e.g., providing feedback according to teaching best practices) and program level (e.g., ensuring that students across semesters meet the same standard). In this paper, we report on the design, development, and introduction of an automated assessment tool to improve instruction. The use of this tool has been aligned with a course on data structures & algorithms taught at the sophomore level. We discuss the development of the tool, the techniques that it uses, and evaluate its impact on grade accuracy.

This Innovative Practice Full Paper presents an automated reasoning approach to the assessment of a student's programming process as captured by an autograder. When students are assessed in computing courses, they are typically assessed on their completed work, such as a programming assignment. Although such an assessment measures a student's general ability, it provides only an indirect measure of the effectiveness of their programming (or problem-solving) process. Without the ability to examine the process by which a student completes an assignment, an instructor may have difficulty teaching problem-solving skills since they cannot measure how the student's skill develops. Some computing courses make use of an autograder which automatically assesses student homework. The trace captured by an autograder is a unique source of information about a student's problem-solving ability, which can be analyzed to measure properties of that learner's process. This provides a way to assess the process-based aspect of student learning, enabling instructors to not only teach problem-solving skills but to determine how well students acquire these skills. This enables teachers to evaluate the methodology they employ and to select instructional material based on student need. Automatically analyzing the trace captured by an autograder has two major challenges: the scale of the data makes traditional approaches computationally expensive, and the definition of a problem-solving process is often expressed in a commonsense way based on an expert's intuition. These challenges can be addressed by using an Answer Set Programming (ASP) approach. ASP is a declarative language that is intended for solving computationally hard problems, and through various extensions, supports human-style commonsense reasoning. Assessing a student is computationally hard due to the combinatorial nature of the different ways an assignment may be solved. In this paper, we discuss the design of an ASP-based system for automatically assessing a student's problem-solving process using the trace captured by an autograder, and present a preliminary evaluation of the system applied within a university-level course on data structures & algorithms.

This paper presents stableKanren, a miniKanren extension with normal logic programming support under stable model semantics. MiniKanren is a relational programming solver implemented atop Scheme via shallow embedding, which means the predicate in each rule is encoded as a goal function directly. The solver utilizes the pattern matching macro in Scheme to transform the input goal function and form a static search stream through continuations to achieve the essential features, resolution and unification, in Prolog. However, the static stream only works on monotonic reasoning. Even though the core miniKanren is designed to be easily modified and extended with new features, none of the existing extensions support solving normal logic programs. Also, no normal logic program solvers are based on a functional programming language. We identify and categorize the roles of resolution and unification in top-down solving. And we realize that a dynamic search stream is needed to support non-monotonic reasoning. So we evolve both resolution and unification with new roles, and we exploit the advantages of using macros and continuations further to weave the information generated during runtime into future streams dynamically. We create multiple innovative macros to compile the normal logic program into a program with its complement form, obtain the domain of a variable under different contexts, and generate the new search stream during solving. And we use the coinductive resolution to handle the loop in the normal logic program. In future work, we plan to apply bottom-up optimization to improve our top-down system performance and support various input rules.

Calculus as a math course is an important subject students need to succeed in, to venture into STEM majors. The paper focuses on the early detection of at-risk students in a calculus course which can provide the proper intervention that might help them succeed. Calculus has high failure rates which corroborate with the data collected from our University that shows us that 40% of the 3266 students whose data were used failed in their calculus course. Some existing studies similar to our paper make use of open-scale data that are lower in data count and perform predictions on low-impact MOOC-based courses. Paper proposes, an automatic detection method of academically at-risk students by using Learning Management Systems (LMS) activity data along with the student information system (SIS) data from our University for the Math course. The proposed method will detect students at risk by employing machine learning to identify key features that contribute to the success of a student. The model developed has a predictive accuracy of 73.5 % on the online modality of the Math course and has 87.8 % accuracy on the face-2-face (F2F) modality of the same class. Transfer student, a binary feature attributed to the highest feature importance.

This Innovative Practice Full Paper presents the experience of using formative data produced by an autograder tool to understand how students complete assignments in a second year computing course. Autograding has increasingly been used in computing courses to evaluate student work. Using autograders supports the scalability of classes and provides other advantages such as accurate assessment, reproducibility over semesters, and rapid feedback. As a consequence of using an autograder, students produce a trace of their problem-solving process while completing assignments. In contrast to the summative assessment performed on a final submission, this trace captures the formative steps that were involved in the development of the final submission. Analyzing this trace can provide insights into student problem-solving methodology as well as the structure of the problems being assigned, and the impact of classroom interventions during an assignment. To produce this view into student learning, we have developed an autograder with the initial goal of improving student instruction. It uses a combination of static and dynamic analysis techniques to support accurate assessment and the ability to check for more complex requirements than can be captured by approaches that compare program output with a ground truth. The tool has been used in a sophomore level course on data structures & algorithms. This course uses the Java programming language and introduces topics such as lists, searching, sorting, binary search trees, priority queues, hash tables, and graphs. Our current analysis work focuses on analyzing traces produced by students to identify difficult parts of an assignment (as represented by grading regressions), and the path through possible solution states that students take as they complete an assignment. In this paper, we discuss the design of this tool, what insights the data captures provide into student learning, and give ideas for future directions supported by gathering formative assessment data.

Imperative and object-oriented programming languages are among the most common languages for general-purpose programming. These languages work well for handling many common tasks necessary for most applications. However, there are still many hard problems that remain difficult to implement directly in imperative languages. Declarative languages have worked well for solving many of these problems by providing a syntax that allows the user to focus on modeling the problem rather than on designing an algorithm. Logic programming languages, like Prolog, have seen success in constraint satisfaction problems, logical databases, and various NP-Hard problems. Unfortunately, these languages have not seen the same success in general-purpose programming, and most of the problems they solve do not exist in isolation. Furthermore, many imperative programmers are still unfamiliar with and unaware of logic programming.

Availability of affordable image and video capturing devices as well as rapid development of social networking and content sharing websites has led to the creation of new type of content, Social Media. Any system serving the end user’s query search request should not only take the relevant images into consideration but they also need to be divergent for a well-rounded description of a query. The previous state-of-the-art methods used a number of views of a particular image as one of the key parameters in order to achieve diverse results. This parameter, while improving the overall results, can omit images that are most recently taken. This might not work if the user is interested in recent images when solving the problem of divergence. Secondly, all prior work considered only one of the clustering algorithms for diversification. The performance of most of these clustering techniques is highly data-dependent and might render inefficient for different kinds of datasets. The main focus of this paper is to use visual description of a landmark location by choosing diverse pictures that best describe all the details of a queried location from community-contributed data sets. For this, an end-to-end framework has been built, to retrieve relevant results that are also diverse. Different retrieval re-ranking and diversification strategies are evaluated to find a balance between relevance and diversification. Clustering techniques are employed to improve divergence. A unique fusion approach has been adopted to overcome the dilemma of selecting an appropriate clustering technique and the corresponding parameters, given a dataset to be investigated. Extensive experiments have been conducted on the Flickr Div150Cred dataset. This system has proved to achieve results that are on par with the start-of-art work done on the MediaEval Challenge. This is achieved without using one of the key parameters that contribute to the improved overall metric results - "Number of Views".

We propose a new practice for the instruction of problem-solving skills: the Analysis-Design-Justification (ADJ) framework. The ADJ framework consists of learning outcomes which represent problem-solving skills, a type of problems to assess those outcomes, and a three-step process for problem-solving. Problems are open-ended and ill-defined, like those in the real world. Although there are several approaches to instructing problem-solving, ADJ supports a course environment with four specific requirements: established syllabi (ADJ may be added to a class already covering many topics), scale (ADJ can handle large classes), modality (ADJ can be performed synchronously and asynchronously), and heterogeneity (ADJ helps to provide scaffolding for diverse student populations). This framework takes inspiration from Polya's problem-solving process and uses the theoretical foundations underlying Problem-based Learning (PBL). The ADJ framework was enacted in the fall 2019 section of a Software Engineering (SE) class on operating-systems at Arizona State University (ASU). In the class, students were introduced to the ADJ framework through a series of group activities, and then asked to solve three ADJ problem sets. Students responded positively to use of the ADJ framework as both an instructional tool for class material, and to mature their problem-solving skills. However, some students had issues carrying out the ADJ process due to its focus on justification as opposed to constructing designs.

This Innovative Practice Work in Progress presents using the concept of a simulated internship to transition students to problems (and environments) which resemble those found in industry or research. New engineering graduates struggle not just with the lack of technical skills specific to a company but have issues in generalizing their existing skill set. They also lack procedural knowledge that is contextual to industry (e.g., dealing with stakeholders, communicating solutions). This requires an educational setting that is more conducive to practical problems than a typical course - students already acquire the needed technical skills but instead need to practice applying and integrating them. This can be supported with the creation of a simulated internship (SimInt) environment that aims to enable development and assessment of outcomes that more accurately capture solving real world problems than current technical skill outcomes. Although students who can complete an internship are already able to mature their skills, some students, for many reasons, are unable to pursue such opportunities. This can be addressed with SimInt. We propose a course to mimic the environment of an internship: students would be assigned a large-scale problem that is open-ended and ill-defined and solve the problem in the context of a simulated company. The focus of this experience would be on maturing problem-solving skills rather than practicing specific technical skills. This would involve integrating aspects of process (e.g., applying with a resume, interviewing, etc.), the environment of working with different stakeholders, and the development of solutions. In this work, we contribute a model for a real-world work environment that captures aspects not found in a typical classroom environment. We describe the differences between typical content-based course (e.g., lecture, flipped) and the new model. The model is described in terms of system elements and interactions between them that are conducive to student learning. SimInt is intended to offer an experience similar to a capstone while supporting scalability and being more appropriate as a formative experience.

Plagiarism is a huge problem in a learning environment. In programming classes especially, plagiarism can be hard to detect as source codes' appearance can be easily modified without changing the intent through simple formatting changes or refactoring. Many source code plagiarism tools do not support a high number of languages because doing so requires maintaining too large of a codebase. It is also difficult to add support for new languages because each language can be vastly different syntactically. Tools that are more extensible often do so by reducing the features of a language that are encoded and end up closer to text comparison tools than structurally aware program analysis tools [27]. This paper introduces a new tool called Kitsune, a plagiarism detection tool, focused on syntactically and structurally aware yet adaptable plagiarism detection. Kitsune has been evaluated for 10 of the languages in the Antlr4 grammar repository with success and could easily be extended to support all the grammars currently developed by Antlr4 or future grammars which are developed as new languages are written.

Problem solving is a necessary skill for computing students to succeed in industry. Students commonly find it difficult to apply their technical skills in solving real-world problems, which can be addressed by the inclusion and scaffolding of open-ended and ill-defined problems in formative computing courses. An Analysis, Design, and Justification (ADJ) based framework was recently developed to enable this goal. The ADJ framework consists of learning outcomes, a type of problem, and a three-step problem-solving process. ADJ supports instruction in environments with constraints (such as large classes, and heterogeneous student ability) that are difficult to support with other approaches (such as Problem-Based Learning). In this paper, we report the experience and impact of leveraging the ADJ framework to develop real-world problem-solving skills in the context of a formative computing course. The ADJ framework was enacted in a class on operating-systems taught at the junior level. In the class, students were taught the concepts of the ADJ framework using a series of discussion activities, and then asked to apply it to generate solutions to three problem sets. The problem sets contained problems on topic areas covered by other assessments in the course but used open-ended and ill-defined problems. Assessments showed that students had different performance on existing (well-defined) problems as opposed to the real-world style problems of ADJ. The ADJ assessments provided a step towards better evaluation of problem-solving ability.

Online courses have gained popularity and seen a surge in enrollment with a reported 58 million students. Adaptive learning is an educational method that is applicable to online courses. Computers adapt the presentation of educational material according to students' learning needs. One of the major challenges with existing systems is that learners are not able to keep up with the instructions in the course that leads to a very low course completion rate. Personalization of the course materials based on the needs of a student is of great value. We propose an adaptive framework for learning that groups students and charts a course plan with the end goal of helping the learner complete all topics in the course. The system also provides feedback about the learner's strong and weak topics with a view to help them learn better. We present a declarative approach that is quite different from existing approaches and provides the user flexibility to specify the constraints and actions as well as consequence of each action instead of having the user encode how to find the solution.

Machine translation (MT) between natural languages is an infamously difficult problem in Natural Language Processing that is still very much being researched. This research study explores the efficacy of developing an adaptive translator using Lexical Functional Grammars. The main research objective is building a machine translator generator for multilingual communication, i.e. developing a system whose inputs are linguistic descriptions of a desired source and target language and whose output is a program that translates between the two natural languages. A bidirectional machine translator between English and Hungarian, developed as a proof-of-concept case study, is discussed. The benefits and drawbacks of this approach as generalized to MT systems are also discussed, along with possible areas of future work.

Answer Set Programming (ASP) represents an elegant way of introducing non-monotonic reasoning into logic programming. ASP has gained popularity due to its applications to planning, default reasoning and other areas of AI. However, none of the approaches and current implementations for ASP are goal-directed. In this paper we present a technique based coinduction that can be employed to design SLD resolution-style, goal-directed methods for executing answer set programs. We also discuss advantages and applications of such goal-directed execution of answer set programs, and report results from our implementation.

The success and acceptance of Web service composition depends on computing solutions comprised of trustworthy services. In this paper, we extend our Web service Composition framework to include selection and ranking of services based on their reputation score. With the increasing popularity of Web-based Social Networks like Linked in, Facebook, and Twitter, there is great potential in determining the reputation score of a particular service provider using Social Network Analysis. We present a technique to calculate a reputation score per service using centrality measure of Social Networks. We use this score to produce composition solutions that consist of services provided by trust-worthy and reputed providers.

Service-oriented computing (SOC) has emerged as the eminent market environment for sharing and reusing service-centric capabilities. The underpinning for an organization's use of SOC techniques is the ability to discover and compose Web services. Although industry approaches to composition have a strong notion of business processes, these approaches largely use syntactic descriptions. As such composition is limited since the true functionality of ambiguous service operations cannot be inferred. Alternatively, academia uses semantic approaches to disambiguate services, but, at the same time, most of these approaches neglect the process rigor needed for complex compositions. In this paper we present a generalized semantics-based technique for automatic service composition that combines the rigor of process-oriented composition with the descriptiveness of semantics. Our generalized approach extends the common practice of linearly linked services by introducing the use of a conditional directed acyclic graph (DAG) where complex interactions, containing control flow, information flow and pre/post conditions, are effectively represented. Furthermore, the composition can be represented semantically as OWL-S documents. Our contributions are applied for automatic workflow generation in context of the currently important bioinformatics domain.

Service-oriented computing is gaining wider acceptance. For Web services to become practical, an infrastructure needs to be supported that allows users and applications to discover, deploy, compose and synthesize services automatically. For this automation to be effective, formal semantic descriptions of Web services should be available. In this paper we formally define the Web service discovery and composition problem and present an approach for automatic service discovery and composition based on semantic description of Web services. We also report on an implementation of a semantics-based automated service discovery and composition engine that we have developed. This engine employs a multi-step narrowing algorithm and is efficiently implemented using the constraint logic programming technology. The salient features of our engine are its scalability, i.e., its ability to handle very large service repositories, and its extremely efficient processing times for discovery and composition queries. We evaluate our engine for automated discovery and composition on repositories of different sizes and present the results.

In a short period the Web has become an important part of our lives. However, the full potential of the Web is still not realized. Two recent developments - Web services and the semantic Web - are steps in the direction of utilizing the full potential of the Web. Web services allow applications to utilize the Web for automatically extracting (and updating) information while the semantic Web enterprise promises to provide the infrastructure that allows intelligent Web services to be rapidly created and deployed. However, with this comes the task of transforming the traditional Web-based systems to Web-services over the semantic Web. In this paper, we demonstrate how an existing successful Web-based system for providing help to first responders of chemically hazardous emergencies (called E-plan) can be converted into a Web-services based model using the semantic Web and intelligent reasoning technologies. Our efforts can be regarded as a case study in converting monolithic Web-based applications to a more agile, rapidly deployable intelligent Web-services model.

To fully utilize Web-services, users and applications should be able to discover, deploy, compose and synthesize services automatically. This automation can take place only if a formal semantic description of the Web-services is available. In this paper we present a markup language called USDL (Universal Service Description Language), for formally describing the semantics of Web-services.