In this paper, we tackle the problem of finding individual equations in students' handwritten assignments. Mathematical equations are the most abundant type of content in solutions to free-response problems in various academic subjects, particularly engineering and science. Finding the individual equations within a solution is an important step toward automated grading. We formulate this task as stroke classification and grouping problems. In the stroke classification problem, we classify each stroke as an equation or non-equation stroke. Stroke grouping is the task of identifying the equation strokes that belong to each equation. In this work, we present a GRU-CRF model that considers both intrinsic properties of the strokes and features computed from pairs of chronologically consecutive strokes. The method simultaneously classifies and groups strokes by computing labels for them. A chaining process is used to combine consecutively drawn equation strokes into equation groups. A binary classifier is then used to combine equation segments comprising single equations that were not written contiguously. We report the performance of our approach on a dataset of 1,060 pages written by 132 students that consists of 9,949 equations. Our approach can find 54% of the equations perfectly and finds an additional 16% with only a few mistakes.

Automatic semantic labeling of strokes in online handwritten documents is a crucial task for many applications such as diagram interpretation, text recognition, and search. We formulate this task as a stroke classification problem in which each stroke is classified as a cross-out, free body diagram, or text. Separating free body diagram and text in this work is different than the traditional text/non-text separation problem because these two classes contain both text and graphics. The text class includes textual notes, mathematical symbols/equations, and graphics such as arrows that connect other elements. The free body diagram class also contains graphics and various alphanumeric characters and symbols that mark or explain the graphical objects. In this work, we present a novel deep neural network model for classification of strokes in online handwritten documents. There are two input sequences to the network. The first sequence contains the trajectories of the pen strokes while the second contains features of the strokes. Each of these sequences is fed to its own CNN-BLSTM channel to extract features and encode relationships between nearby strokes. The output of the two channels is concatenated and used as the input to a CRF layer that predicts the best sequence of labels for given input sequences. We evaluated our model on a dataset of 1,060 pages written by 132 students in an undergraduate statics course. Our model achieved an overall classification accuracy of 94.70% on this dataset.

This study aims to identify procrastinating behavior of students by tracking their activities in an undergraduate engineering course and to explore its relationship with course performance. In our research, we have examined methods for instrumenting students' ordinary learning activities in undergraduate engineering courses. More specifically, we provide students with digital smartpens they use to complete their assignments, lecture notes, and exams. We also provide them with an instrumented document viewer that records reading effort for course documents. This technology provides a fine-grained, time-stamped record of writing and reading throughout the course. In the present work, we examine a student's studying effort in relation to the due dates of assignments and compare this with academic achievement in a course. The results show that when a student studies is a powerful predictor of academic success. For example, students who do their work in the 24 hours immediately preceding a due date tend to do poorly, while those who do their work one or two days early, tend to perform well. This work is a first step toward building tools that allow students to monitor their learning habits and use this information to improve their own academic performance.

In many academic subjects, especially science, engineering, and math, paper-based problem solving is an important part of education. However, grading such work can be prohibitively expensive in large university courses. As a remedy, we have developed techniques to support the automated grading of handwritten problem solutions. Students complete their work using Livescribe digital pens and draw boxes around the final answers. We developed techniques that identify the answers by locating the boxes. This problem is challenging as the written work contains a mixture of diagrams and equations, and boxes frequently appear as a part of the diagrams. Additionally, the boxes must be segmented from the remainder of the writing. Thus, a simple shape recognizer is inadequate for this task. Our techniques efficiently locate answer boxes within a complex page of free-form writing. Furthermore, our techniques are designed to be robust to the wide range of variations in the way students write. In a test on 2022 pages of homework problems, our techniques correctly located 95.3% of the 4473 answer boxes. These techniques are an important step towards automated grading of handwritten work because once the answer boxes are located, a variety of handwriting recognition methods can be used to interpret the answers.

Objects in freely-drawn sketches often have no spatial or temporal separation, making object recognition difficult. We present a two-step stroke-grouping algorithm that first classifies individual strokes according to the type of object to which they belong, then groups strokes with like classifications into clusters representing individual objects. The first step facilitates clustering by naturally separating the strokes, and both steps fluidly integrate spatial and temporal information. Our approach to grouping is unique in its formulation as an efficient classification task rather than, for example, an expensive search task. Our single-stroke classifier performs at least as well as existing single-stroke classifiers on text vs. nontext classification, and we present the first three-way single-stroke classification results. Our stroke grouping results are the first reported of their kind; our grouping algorithm correctly groups between 86% and 91% of the ink in diagrams from two domains, with between 69% and 79% of shapes being perfectly clustered.

Sim-U-Sketch is an experimental sketch-based interface we developed for Matlab®'s Simulink® software package. With this tool, users can construct functional Simulink models simply by drawing sketches on a computer screen. To support iterative design, Sim-U-Sketch allows users to interact with their sketches in real time to modify existing objects and add new ones. The system is equipped with a domain-independent, trainable symbol recognizer that can learn new symbols from single prototype examples. This makes our system easily extensible and customizable to new domains and unique drawing styles.

A long standing challenge in pen-based computer interaction is the ability to make sense of informal sketches. A main difficulty lies in reliably extracting and recognizing the intended set of visual objects from a continuous stream of pen strokes. Existing pen-based systems either avoid these issues altogether, thus resulting in the equivalent of a drawing program, or rely on algorithms that place unnatural constraints on the way the user draws. As one step toward alleviating these difficulties, we present an integrated sketch parsing and recognition approach designed to enable natural, fluid, sketch-based computer interaction. The techniques presented in this paper are oriented toward the domain of network diagrams. In the first step of our approach, the stream of pen strokes is examined to identify the arrows in the sketch. The identified arrows then anchor a spatial analysis which groups the uninterpreted strokes into distinct clusters, each representing a single object. Finally, a trainable shape recognizer, which is informed by the spatial analysis, is used to find the best interpretations of the clusters. Based on these concepts, we have built SimuSketch, a sketch-based interface for Matlab's Simulink software package. An evaluation of SimuSketch has indicated that even novice users can effectively utilize our system to solve real engineering problems without having to know much about the underlying recognition techniques.