Tables, ubiquitous in data-oriented documents like scientific papers and financial statements, organize and convey relational information. Automatic table recognition from document images, which involves detection within the page, structural segmentation into rows, columns, and cells, and information extraction from cells, has been a popular research topic in document image analysis (DIA). With recent advances in natural language generation (NLG) based on deep neural networks, data-to-text generation, in particular for table summarization, offers interesting solutions to time-intensive data analysis. In this paper, we aim to bridge the gap between efforts in DIA and NLG regarding tabular data: we propose WEATHERGOV+, a dataset building upon the WEATHERGOV dataset, the standard for tabular data summarization techniques, that allows for the training and testing of end-to-end methods working from input document images to generate text summaries as output. WEATHERGOV+ contains images of tables created from the tabular data of WEATHERGOV using visual variations that cover various levels of difficulty, along with the corresponding human-generated table summaries of WEATHERGOV. We also propose an end-to-end pipeline that compares state-of-the-art table recognition methods for summarization purposes. We analyse the results of the proposed pipeline by evaluating WEATHERGOV+ at each stage of the pipeline to identify the effects of error propagation and the weaknesses of the current methods, such as OCR errors. With this research (dataset and code available here1), we hope to encourage new research for the processing and management of inter- and intra-document collections.

Unsupervised anomaly detection and localization in images is a challenging problem, leading previous methods to attempt an easier supervised one-class classification formalization. Assuming training images to be realizations of the underlying image distribution, it follows that nominal patches from these realizations will be well associated between and represented across realizations. From this, we propose Inter-Realization Channels (InReaCh), a fully unsupervised method of detecting and localizing anomalies. InReaCh extracts high-confidence nominal patches from training data by associating them between realizations into channels, only considering channels with high spans and low spread as nominal. We then create our nominal model from the patches of these channels to test new patches against. InReaCh extracts nominal patches from the MVTec AD dataset with 99.9% precision, then archives 0.968 AUROC in localization and 0.923 AUROC in detection with corrupted training data, competitive with current state-of-the-art supervised one-class classification methods. We test our model up to 40% of training data containing anomalies with negligibly affected performance. The shift to fully unsupervised training simplifies dataset creation and broadens possible applications. Code: github.com/DeclanMcIntosh/InReaCh

The increasing importance of videos as a medium for engagement, communication, and content creation makes them critical for organizations to consider for user feedback. However, sifting through vast amounts of video content on social media platforms to extract requirements-relevant feedback is challenging. This study delves into the use of TikTok and YouTube, two widely used social media platforms that focus on video content, in identifying relevant user feedback that may be further refined into requirements using subsequent requirement generation steps. We demonstrate an approach of using videos as a source of user feedback by analyzing audio and visual text, and metadata (i.e., description/title) from 6276 videos of 20 popular products across various industries. We employed state-of-the-art deep learning transformer-based models, and classified 3097 videos consisting of requirements relevant information. We then clustered relevant videos and found multiple requirements relevant feedback themes for each of the 20 products. This feedback can later be refined into requirements artifacts. We found that product ratings (feature, design, performance), bug reports, and usage tutorial are persistent themes from the videos. Video-based social media such as TikTok and YouTube can provide valuable user insights, making them a powerful and novel resource for companies to improve customer-centric development.

Recent advancements in cabled ocean observatories have increased the quality and prevalence of underwater videos; this data enables the extraction of high-level biologically relevant information such as species’ behaviours. Despite this increase in capability, most modern methods for the automatic interpretation of underwater videos focus only on the detection and counting organisms. We propose an efficient computer vision- and deep learning-based method for the detection of biological behaviours in videos. TempNet uses an encoder bridge and residual blocks to maintain model performance with a two-staged, spatial, then temporal, encoder. TempNet also presents temporal attention during spatial encoding as well as Wavelet Down-Sampling pre-processing to improve model accuracy. Although our system is designed for applications to diverse fish behaviours (i.e, is generic), we demonstrate its application to the detection of sablefish (Anoplopoma fimbria) startle events. We compare the proposed approach with a state-of-the-art end-to-end video detection method (ReMotENet) and a hybrid method previously offered exclusively for the detection of sablefish’s startle events in videos from an existing dataset. Results show that our novel method comfortably outperforms the comparison baselines in multiple metrics, reaching a per-clip accuracy and precision of 80% and 0.81, respectively. This represents a relative improvement of 31% in accuracy and 27% in precision over the compared methods using this dataset. Our computational pipeline is also highly efficient, as it can process each 4-second video clip in only 38ms. Furthermore, since it does not employ features specific to sablefish startle events, our system can be easily extended to other behaviours in future works.

This paper addresses bioprosthetic heart valve (BHV) durability estimation via computer vision (CV)-based analyses of the visual symmetry of valve leaflet motion. BHVs are routinely implanted in patients suffering from valvular heart diseases. Valve designs are rigorously tested using cardiovascular equipment, but once implanted, more than 50% of BHVs encounter a structural failure within 15 years. We investigate the correlation between the visual dynamic symmetry of BHV leaflets and the functional symmetry of the valves. We hypothesize that an asymmetry in the valve leaflet motion will generate an asymmetry in the flow patterns, resulting in added local stress and forces on some of the leaflets, which can accelerate the failure of the valve. We propose two different pair-wise leaflet symmetry scores based on the diagonals of orthogonal projection matrices (DOPM) and on dynamic time warping (DTW), computed from videos recorded during pulsatile flow tests. We compare the symmetry score profiles with those of fluid dynamic parameters (velocity and vorticity values) at the leaflet borders, obtained from valve-specific numerical simulations. Experiments on four cases that include three different tricuspid BHVs yielded promising results, with the DTW scores showing a good coherence with respect to the simulations. With a link between visual and functional symmetries established, this approach paves the way towards BHV durability estimation using CV techniques.

Marine vessel traffic is one of the main sources of negative anthropogenic impact upon marine environments. The automatic identification of boats in monitoring images facilitates conservation, research and patrolling efforts. However, the diverse sizes of vessels, the highly dynamic water surface and weather-related visibility issues significantly hinder this task. While recent deep learning (DL)-based object detectors identify well medium- and large-sized boats, smaller vessels, often responsible for substantial disturbance to sensitive marine life, are typically not detected. We propose a detection approach that combines state-of-the-art object detectors and a novel Detector of Small Marine Vessels (DSMV) to identify boats of any size. The DSMV uses a short time series of images and a novel bi-directional Gaussian Mixture technique to determine motion in combination with context-based filtering and a DL-based image classifier. Experimental results obtained on our novel datasets of images containing boats of various sizes show that the proposed approach comfortably outperforms five popular state-of-the-art object detectors. Code and datasets available at https://github.com/tunai/hybrid-boat-detection.

This paper explores a novel application of document generation: the automatic creation of residential electrical plans from architectural data. Electrical plan documents, crucial to all residential construction and renovation projects, are currently generated manually by electrical designers who must adhere to the local electrical code and follow industry best practices. The designers decide the type and location of all household electrical devices and outlets based on the architectural floor plans. This is a tedious, highly repetitive, and time-consuming task. We propose a procedural approach to automate the generation of residential electrical plans via a stack-based finite state machine model that mimics the electrical designer's thought process. The system receives 2D architectural data (e.g. wall location) as input and yields a customized electrical plan as output. Experimental results on a variety of architectural layouts of bathrooms, bedrooms, and kitchens are very promising and demonstrate the approach's functionality and usefulness. This paper paves the way for new algorithmic tools facilitating the design cycle of building projects.

This paper provides a comprehensive comparative study of traditional, hybrid, and deep learning (DL) methods for detecting marine species in echograms. Acoustic backscatter data obtained from multi-frequency echosounders is visualized as echograms and typically interpreted by marine biologists via manual or semi-automatic methods, which are time-consuming. Challenges related to automatic echogram interpretation are the variable size and acoustic properties of the biological targets (marine life), along with significant inter-class similarities. Our study explores and compares three types of approaches that cover the entire range of machine learning methods. Based on our experimental results, we conclude that an end-to-end DL-based framework, that can be readily scaled to accommodate new species, is overall preferable to other learning approaches for echogram interpretation, even when only a limited number of annotated training samples is available.

A back-of-the-napkin idea is typically created on the spur of the moment and captured via a few hand-sketched notes on whatever material is available, which often happens to be an actual paper napkin. This paper explores the preservation of such back-of-the-napkin ideas. Hand-sketched notes, reflecting those flashes of inspiration, are not limited to text; they can also include drawings and graphics. Napkin backgrounds typically exhibit diverse textural and colour motifs/patterns that may have high visual saliency from a low-level vision standpoint. We thus frame the extraction of hand-sketched notes as a background modelling and removal task. We propose a novel document background model based on texture mixtures constructed from the document itself via texture synthesis, which allows us to identify background pixels and extract hand-sketched data as foreground elements. Experiments on a novel napkin image dataset yield excellent results and showcase the robustness of our method with respect to the napkin contents. A texture-based background modelling approach, such as ours, is generic enough to cope with any type of hand-sketched notes.

This paper addresses the problem of semantic overlap across document objects in the context of ground truth representation for document layout analysis. Document object categories often share primitives from a low-level perspective (e.g. regions inside bars in a bar chart resemble background), making it difficult to evaluate document layout segmentation methods based on pixel classification, as most datasets and ground truth models focus on document objects. We propose a novel ground truth model that utilizes structural and statistical pattern recognition concepts. Statistical pixel-based data derived from low-level elemental patterns are layered onto high-level structural object-based data. We also present evaluation metrics that take advantage of the layered ground truth model, allowing a contextual evaluation of pixel classification algorithms. We apply the proposed model to two recent pixel classification approaches, evaluated on business document images that exhibit a challenging mixture of textual, graphical, and pictorial elements through varied layouts. The proposed model allows to obtain very detailed, comprehensive, and intuitive information on the strengths and limitations of the evaluated approaches that would be impossible to obtain through other models.

Nonverbal cues constitute a significant part of human communication. Traditionally the object of psychology, nonverbal communication studies now permeate fields such as social signal processing and human computer interaction. The ubiquity of digital recordings of human social interactions and of free sharing platforms offers many opportunities for the automated analysis of group interaction dynamics; yet, most research relies on multimodal cues and strict setups, which are incompatible with this vast pool of video data. In this paper, we focus on the automatic identification of non-talking participants in videos of panel conversations acquired in uncontrolled environments, based solely on visual nonverbal cues. Our approach characterizes human body motion with a novel feature descriptor based on a non-linear model of pixel change history; motor behavioral patterns derived from this descriptor are then utilized via supervised machine learning to identify non-talking participants in each frame and provide an assessment of the participants' engagement levels. Performance evaluation on a challenging dataset demonstrated the effectiveness of our approach to detect non-speakers, with an overall F-score of 86.2%, as well as its robustness to varied settings. To the best of our knowledge, this is the first attempt at identifying non-talking participants for engagement level assessments from a computer vision viewpoint, which has several relevant applications, such as content-based video retrieval and video summarization.

This paper focuses on video summarization of abnormal behavior for remote invigilation of online exams. While the last decade has seen a massive increase in e-learning and online courses offered at postsecondary institutions, preserving the integrity of online examinations still heavily relies on web video conference invigilation performed by a remote proctor. Live remote invigilation is limited in the number of students that can be handled at once, and manual post-exam review is labor intensive. We propose a novel computer vision-based video content analysis system for the automatic creation of video summaries of online exams to assist remote proctors in post-exam reviews. The proposed method models normal and abnormal student behavior patterns using head pose estimations and a semantically meaningful two-state hidden Markov model. Video summaries are created from detected sequences of abnormal behavior. Experimental results are promising and demonstrate the viability of the proposed approach, which could readily be expanded to generate real-time alerts for live remote invigilation.

We propose and evaluate a method for learning deep-sea substrate types using video recorded with a remotely operated vehicle (ROV). The goal of this work is to create a labelled spatial map of substrate types from ROV video in order to support biological and geological domain research. The output of our method describes the mixtures of geological features such as sediment and types of lava flow in images taken at a set of points chosen from an ROV dive. The main contribution of this work is the assembly of a pipeline combining several unique approaches which is able to robustly generate substrate type mixtures under the varying lighting and perspective conditions of deep-sea ROV dive videos. The pipeline comprises three main components: sampling, in which a trained classifier and spatial sampling is used to select relevant frames from the dataset; feature extraction, in which the improved local binary pattern descriptor (ILBP) is used to generate a Bag of Words (BoW) representation of the dataset; and topic modelling in which a variant of Latent Dirichlet Allocation (LDA), is used to infer the mixture of substrate types represented by each BoW. Our method significantly outperforms techniques relying on keypoint based features rather than texture based features, and k-means rather than LDA, demonstrating that our proposed pipeline accurately learns and identifies visible substrate types.

In order to safely operate sailboats, captains often rely on proper interpretation of several marine aspects to make decisions. In this project, we have developed an Augmented Reality System (ARS) to provide captains of sailboats with a centralized sensor data server and a visualization method. We have deployed an experimental proof-of-concept version of this system on our research vessel, SV Moon shadow. Assistance in navigation is of particular interest for small sailing vessels as they are sometimes sailed by the captain alone. At the same time there are a large number of data inputs such as wind, tide, weather, position, and presence of obstacles such as logs or kelp that have to be considered to choose the proper course of action. We introduce a visualization tool that provides an interface for representing a wide spectrum of relevant marine data. The interface relies on a real-time data server that provides information about the status of the vessel (wind, GPS, gyro, accelerometer, depth sounder etc.) An important component of the interface is a debris detector that analyzes data from a camera mounted on the bow in order to warn a captain about a potential collision. We have also examined initial feedback on this tool from a number of users.

This paper proposes an approach for the automatic detection and classification of changes occurring in images of documents with identical content, but generated with different software versions, or under different operating platforms. Our work is performed on a database of digitally-born business documents created using financial reporting tools. The proposed method involves a multi-stage process, where the end goal is to present to a human user the reports which have changed and the changes which were detected. Our main contribution is related to matching and comparing of graphical document elements. This paper focuses on detection of local, translation-based changes. Future work will explore other local changes involving size, color, and rotation.

In this paper, we present a challenging dataset for the purpose of segmentation and change detection in photographic images of mountain habitats. We also propose a baseline algorithm for habitats segmentation to allow for performance comparison. The dataset consists of high resolution image pairs of historic and repeat photographs of mountain habitats acquired in the Canadian Rocky Mountains for ecological surveys. With a time lapse of 70 to 100 years between the acquisition of historic and repeat images, these photographs contain critical information about ecological change in the Rockies. The challenging aspects of analyzing these image pairs come mostly from the perspective (oblique) view of the photographs and the lack of color information in the historic photographs. The baseline algorithm that we propose here is based on texture analysis and machine learning techniques. Classifier training and results validation are made possible by the availability of expert manual ground-truth segmentation for each image. The results obtained with the baseline algorithm are promising and serve as a reference for new and improved segmentation and change detection algorithms.

This letter addresses the texture classification problem through a pixel-based local binary pattern (LBP) statistics aggregation mechanism. Real-world texture images often present challenges for classification algorithms in terms of intra-class variability due, among others, to variable illumination. The LBP operator, a state-of-the-art texture descriptor, possesses key properties for tackling real-world texture images: discriminative power and invariance against monotonic gray level changes. We propose a novel texture classification approach that increases the robustness of LBP-based methods with respect to any type of intra-class variations. The method locally characterizes each pixel with an LBP code histogram and globally computes the label of a textured image by aggregating pixel labels through a voting process. Our approach can be in principle applied to any LBP version, as it focuses on how statistics are computed from LBP codes. We show that the proposed pixel-based approach improves upon traditional LBP block-based approaches in terms of classification accuracy by up to 5.1 p.p. on the public Outex database for the classic LBP with various neighborhoods as well as for various LBP extensions.

This paper exploits the concept of sparseness to generate novel contextual multi-resolution texture descriptors. We propose to extract low-dimension features from Gabor-filtered images by considering the sparseness of filter bank responses. We construct several texture descriptors: the basic version describes each pixel by its contextual textural sparseness, while other versions also integrate multi-resolution information. We apply the novel low-dimension sparseness-based descriptors to the problem of texture segmentation and evaluate their performance on the public Outex database. The sparseness-based descriptors show a substantial improvement over Gabor filters with respect not only to computational costs and memory usage, but also to segmentation accuracy. The proposed approach also shows a desirable smooth, monotonic behavior with respect to the dimensionality of the descriptors.

The free on-line Artificial Intelligence (AI) course taught by Norvig and Thrun in Fall 2011 is likely to be a game changer in postsecondary education. The huge course enrollment demonstrates that there is a keen interest in this new type of open education, and that no campus-based alternative can be suitable for educating such large numbers of students. The AI course was offered as a sequence of granular, interactive video clips grouped into topics. Homework and exams were video-based as well. This paper provides a critical evaluation of the author's experience as a student enrolled in the advanced track of the AI course. The author is a tenured associate professor at a Canadian medium-sized, research-intensive university. Her expertise lies in Computer Vision, a topic that is closely related to Artificial Intelligence. The paper is based on participatory action research methods.

Canadian undergraduate and graduate programs in Engineering and Computer Science attract a large number of international students. This is a relatively recent phenomenon with social and academic implications that are not completely understood. We are aware that more can be done for the recruitment, retention, and more generally for increasing the quality of the learning experience of our international students. More efforts need to be made in order to foster and expand social and academic interactions between Canadian and international students, as well as student-faculty interactions. The research described in this paper aims to identify the first steps in creating an inclusive environment that fosters academic, social, and personal growth for both international and Canadian students. This study discusses data collected about the experience of international undergraduate students in the Faculty of Engineering our university. The purpose of the data collection was to determine their specific needs, and to solicit suggestions and recommendations about ways in which to address them.

This paper presents the results of a comparison study using the Visual VIPERS interface, a graphical interface which can be applied as an educational tool for novice computer vision students. The goal of the study was to evaluate the change in usability of the interface after the addition of a monitor tool, which can be used to view intermediate image results at specific stages of an algorithm. A user study was conducted in which participants were asked to find an error in a pre-assembled algorithm. Results indicate that participants using the older version of the interface (with no monitor tool) took, on average, less time to find the error than participants who used the monitor tool. However, interview responses indicated a greater level of understanding of the algorithm from participants who used the monitor tool. Interview responses also demonstrated a clear desire from users of the old interface for the addition of a debugging tool (such as the newly introduced monitor tool). Our belief is that participants using the monitor tool performed a more thorough search of the algorithm, and thus gained a greater understanding of how the algorithm operated, while attempting to determine the source of the error.

This paper proposes a novel approach for monitoring sleep using pressure data. The goal of sleep monitoring is to detect and log events of normal breathing, sleep apnea and body motion. The proposed approach is based on translating the signal data to the image domain by computing a sequence of inter-frame similarity matrices from pressure maps acquired with a mattress of pressure sensors. Periodicity analysis was performed on similarity matrices via a new algorithm based on segmentation of elementary patterns using the watershed transform, followed by aggregation of quasi-rectangular patterns into breathing cycles. Once breathing events are detected, all remaining elementary patterns aligned on the main diagonal are considered as belonging to either apnea or motion events. The discrimination between these two events is based on detecting movement times from a statistical analysis of pressure data. Experimental results confirm the validity of our approach.

This paper proposes a method to obtain fronto-parallel (side-view) body part trajectories for a walk observed from an arbitrary view. The method is based on homography transformations computed for each gait half-cycle detected in the walk. Each homography maps the body part trajectories to a simulated side view of the walk. The proposed method is stable as the resulting normalized trajectories are not influenced by missing or omitted parts of the raw trajectories. Experiments confirm that normalized trajectories are in agreement with the ones that would be obtained from a side view.

This paper discusses an interdisciplinary educational initiative led by an instructional team with backgrounds in engineering and psychology in the context of an ergonomics course for software engineers. Our case study evaluates the educational outcomes of a course project that dealt with the error analysis and prototype-level redesign of a software tool for elderly users. The paper presents the rationale for the choice of this project, the project organization, and the evaluation of project-related outcomes with respect to the course learning objectives.