Scanning Electron Microscopy (SEM) is one of the most important imaging techniques to understand the dynamics of microscale objects. However, SEM images could mostly produce two-dimensional images which limit the exploration of volumetric information. To address this limitation, we propose the integration of a state-of-the-art machine learning approach that leverages Confocal Laser Scanning Microscopy (CLSM) image data for training a deep learning model. The objective of this model is to approximate volumetric information by taking a 2D SEM image as input and generating corresponding depth information. The depth of information then be utilized to estimate volumetric quantification such as biofilm density and bacteria cell counts.

Supervised deep learning techniques have demonstrated remarkable performance in segmentation tasks in various domains including medical, biomaterial, and bioengineering fields. The advancement in imaging technologies has enabled the gathering of large amounts of raw data. Nevertheless, acquiring fully annotated ground-truth datasets for supervised deep learning-based segmentation tasks has remained a challenge attributed to the availability of expert resources and the laborious task of pixel-level annotation. Self-supervised learning(SSL) techniques have gained the momentum to learn representations from unlabeled datasets while also preserving the domain-specific information over transfer learning techniques thereby overcoming the challenge of annotating huge datasets. Weakly supervised learning(WSL) techniques have gained attention to address the annotation challenge by using weak annotations such as scribbles, dot annotations, and bounding boxes instead of pixel-wise annotations. In this work, we leveraged the advantages of both SSL and WSL to perform segmentation of single cells in optical images of biofilms attached on single-layer graphene-coated copper substrates guided by scribble annotations. Conditional random fields are applied as a post-processing to further improve segmentation performance. The method showed promising performance in segmenting single cells on biofilm images with scribble annotations that make up only 40-50% of pixel-level annotations.

Biofilms are ubiquitous in aqueous environments, exerting significant influence on diverse surfaces, including metals prone to microbiologically influenced corrosion (MIC). This multifaceted phenomenon demands interdisciplinary collaborations to combat its far-reaching implications. In this context, our research delves into the intricate characterization of twodimensional (2D) materials, particularly hexagonal boron nitride (hBN), which is crucial for advancing corrosion prevention coatings. The nanoscale dimensions of 2D materials pose challenges in microstructural analysis and defect identification, necessitating labor-intensive traditional techniques. To address these complexities, we utilized two unsupervised machine learning models, namely, (a) K-means clustering, and (b) Gaussian Mixture Model (GMM), which enabled clear differentiation between multilayer hBN (MLhBN) and cracks. Our approach will streamline the characterization process and facilitate the extraction of thin layers with enhanced accuracy.

Serious games have been widely used as medium of instruction in various domains. A serious game is composed of various exercises a player has to accomplish in order to achieve the final goal. Each exercise is designed using a set of concepts that a player has to achieve proficiency in. The design of problems is of greater importance in such scenarios than the physical game design. However, traversal through the game is challenging for users with no gaming experience. Spatial distribution of these exercises in the game may influence the learning potential of a player, especially when the exercises have a conceptual dependency associated with them. Our approach in this project is to map the problem graph on to a selected environment in the game for efficient traversal through the game space. Towards this goal, we plan to explore existing environments from Unreal Engine with possible locations for exercises. We will study the theory of graph embedding used to map multi-threaded programs to high performance architecture topologies and explore their adaptation to the problem of exercise distribution in serious game environments.

This Work in Progress outlines a framework of a new intelligent e-learning platform for quantum cybersecurity education. The platform uses intelligent notebooks to support multiple modes of learning through a rich set of media that includes text, videos, interactive widgets, simulations and can even incorporate serious games. The new platform, named Quark, is designed for undergraduate and graduate students, professionals and self-learners wanting to learn the basics of emerging areas in cybersecurity. The Quark platform consists of a learning bank data repository based on FAIR (Findable, Accessible, Inter-operable, and Reusable) principles storing learning objects designed using object oriented (OO) methods and incorporated using contemporary pedagogical methods to support a multiple, varied learning experiences for each topic. The overarching vision of this work is the development of a novel e-learning platform that synthesizes engaging learning experiences so that learners can achieve targeted proficiency in quantum cybersecurity education. In contrast to e-books and other elearning repositories, Quark offers a dynamic way to create subject content satisfying a given set of student-learning objectives for achieving the desired student learning outcomes with high levels of engagement and proficiency. Domain experts drive the content creation enriched with metadata that allows automatic processing of content through algorithms in Quark to synthesize a family of Python-based Jupyter Notebooks and lesson plans. Students select their learning objectives and outcomes, time to completion and student learning choices. The system then dynamically builds the lesson plan based on the dependencies in the metadata defined by the domain experts. This work in progress describes the Quark framework using sample skeleton content. Quark is the first intelligent notebook platform of its kind designed for quantum cybersecurity. In the future, Quark will be able to customize content continuously based on student interactions and informed learning choices and can be set-up for use with any topic.

Scanning electron microscopy (SEM) images play a crucial role in the quantitative analyses of biofilms on materials by providing detailed information about biofilm formation, ultrastructure, cells and their interaction with materials. In addition to some intrinsic limitations of SEM imaging, some of the characteristics of images in SEM volumes often vary in terms of the wide range of magnifications, high resolutions, depth of the field, and the SEM protocols. Quantitative characterization of biofilm morphologies such as the cell size, geometry, and density from these SEM volumes is a challenging problem. This paper presents deep learning based super-resolution (DLSR) as a step towards addressing this problem. Three DLSR approaches based on generative adversarial learning techniques are applied to an SEM biofilm dataset and compared in terms of their a) preservation of the morphological features, and b) their impact on the performance of a deep learning based image segmentation task. Our results show that the DLSR approaches vary in preservation of morphological features. We also show that DLSR can considerably improve the performance of image segmentation outputs of deep learning networks and hence be incorporated in any deep learning pipeline used for quantitative analyses of biofilms based on SEM images.

With the growth of technology capturing huge amounts of image data has become possible including electron microscopic images. Deep learning techniques have been thus drastically improved for analysing images due to their huge availability. Deep learning has been applied in various domains including medical, biomaterial, engineering fields t o analyze complex images. However, supervised deep learning techniques require huge amounts of annotated images. Annotating the images, specifically pixel wise annotations for segmentation tasks could be overwhelming and requires expert resources. Weakly-supervised learning has been popularly employed in such scenarios where weak labels are used for segmentation purposes. Also, self-supervised learning techniques have greatly reduced the amount of labeled data required to train a model for any downstream task. In this study, we would employ self-supervised learning technique followed by scribble supervision for performing biofilm segmentation on optical images. Our initial classification results and the proposed method for segmentation using scribble annotation are provided in this paper.

As biofilms research grows rapidly, a corpus of bibliographic literature (i.e., documents) is increasing at an incredible rate. Many researchers often need to inspect these large document collections, including (1) text, (2) images, and (3) captions, to understand underlying biological mechanisms and make a critical decision. However, researchers have great difficulty in exploring such ever-growing large datasets in labor-intensive processes. Thus, automation of such tasks is urgently required for the automatic identification or classification of a large volume of document collections. To address this problem, we present a multimodal deep learning-based approach to automatically classify documents for a specialized information retrieval technique based on biofilm images, captions, and texts, which is a major source of information for the classification of documents. Images, captions, and texts from biofilm documents are represented in a large vector space. Then, they are fed into convolutional neural networks (CNNs), to improve similarity matching and relevance. Our extensive experiments and analysis will take captions, texts, or images as unimodal models as inputs and concatenate them all into multimodal models. The trained models for this classification approach in turn help a search engine to precisely identify relevant and domain-specific documents from a large volume of document collections for further research direction in biofilm development.

Detection of microbially influenced corrosion (MIC) products in biofilm images is an important problem in material science and engineering. Neural network models that can accurately detect biofilm images having corrosion products are likely to have significant and broad implications in the study and development of materials. However, generating and annotating biofilm image datasets in sufficient volumes is a major impediment in developing such networks. A self-supervised learning approach is presented for automatically detecting MIC products on metal surfaces based on analyses of Scanning Electron Microscope (SEM) biofilm images. The proposed approach uses a Simple Siamese (SimSiam) architecture to learn visual image representations from an unlabeled set of biofilm images, which is then fine-tuned using a scarce set of labeled images to build a model to detect biofilms containing corrosion products. The architecture generates two different augmented versions of the input images and learns the representations using an encoder that uses ResNet backbone. The architecture aims to minimize the negative cosine similarity of the outputs from the encoders and hence learns the representations of the images as both augmented versions belong to the same image. In order to improve the dataset quality and volume, input images are contrast enhanced, scaled, and overlapping image patches are generated and used to learn representations and fine-tuning. The performance of the models are analyzed using precision and recall metrics for patches of varying sizes. An overall accuracy of 62% was obtained for the classification of corrosion in the images after finetuning the model with scarce labeled dataset. Our results show that the models built using the proposed self-supervised learning approach can successfully detect corrosion products in biofilm images and that the performance of the models successively improves with increase in the patch size.

Medical image segmentation is a challenging problem for computer vision approaches where deep learning networks have achieved impressive successes in recent years. In this paper, we propose a directed, fine tuning approach for instance segmentation networks by using feature clustering of predictions along with labeled training instances to improve network performance. The approach directs and limits analyses of predicted instances by experts to similar training instances only and reduces manual overheads by managing the number of instances that need to be examined. Sub-optimal network predictions are handled either by retraining the networks on data augmented with the relevant training instances, correcting training labels, and/or by readjusting network inference parameters. We first develop a state-of-the-art Mask R-CNN based network for instance segmentation of fundus images with retinal lesions and scars caused by Ocular Toxoplasmosis. Then, we show how the proposed approach can be applied to fine tune this network in a directed manner using feature clustering using a pre-trained CNN network. We demonstrate the robustness of our proposed approach with the evaluation results - mask average IoU increased by 7% and mAP under 0.5 IoU threshold increased by 20%. Our experiments also show that fine tuning by analyzing 66% of the predicted instances achieves the same improvement as that obtained by all of the predicted instances, a significant reduction of the manual overheads for fine tuning.

Recent advances in semantic segmentation using deep learning methods have achieved promising results on several benchmark datasets. However, the primary challenge involved in such segmentation approaches is the availability of applicable training data. Since only experts are equipped to effectively annotate (or label) any available data for training semantic segmentation networks, the effort and cost involved can be considerable, especially for larger datasets. In this paper, we aim to address this problem by proposing a Thrifty Annotation Generation approach that records high performance on segmentation networks with minimal expert effort and cost (intervention). We present a deep active learning framework that combines the use of marker-controlled watershed (MC-WS) algorithm to generate pseudo labels for segmentation networks (U-Net) and active learning to significantly minimize effort and cost by selecting only the most impactful training data for labeling. We built the initial U-Net model by generating pseudo labels for the training data using MC-WS. We then make use of the uncertainty information (entropy) of each image provided by the U-Net to determine the most uncertain or effective images for expert labeling. We evaluated the TAG approach using the 2012 ISBI Challenge dataset for 2D segmentation and a novel Biofilm dataset. Our approach achieved promising segmentation accuracy (IoU) and classification accuracy with minimal expert intervention. The results of our experiments also indicate that the TAG approach can be generalized to achieve high-performance segmentation results on any dataset using minimal expert effort and cost.

Ocular Toxoplasmosis (OT) is a widespread infectious chorioretinal disease whose timely diagnosis and treatment are crucial to prevent potential vision loss. Diagnosing OT is a challenging task ranging from tedious analyses of fundus images of the eye to serological clinical tests. An automated approach using convolutional neural networks (CNNs) towards diagnosing OT by analyzing fundus images is described. Fundus images are segmented to patches using a sliding window and are classified into healthy and unhealthy fundus image patches using a CNN model. An OT lesion heat map of a fundus image is generated from these patches. The heat map and patch features are then combined to develop a dual input hybrid CNN model detecting OT fundus images with high accuracy. The approach was applied to a dataset of fundus images involving OT and normal subjects and was highly effective in identifying fundus images having OT lesions.

Integrating of subject matter onto serious games is an important problem that has been shown to impact the learning potential of serious games. A novel approach, inspired by peer-to-peer (P2P) networks, towards designing and deploying a series of problems in game scenarios is described. Given a set of problems involving a set of concepts the proposed approach automatically generates a problem network graph akin to P2P network that can then traversed by a player to collect all the concepts that are necessary to learn a topic of interest. A network traversal algorithm is described, which identifies the relevant problems and produces an efficient route through the network for learning the topic. We also describe an algorithm for mapping the problem network graph onto a game scenario by identifying groups of problems that can be placed in a single location of the game like the level of a building, arcade, or a room, physical barriers that separate, and the conditions for passing through these barriers. The proposed approach has been validated through a quantum cryptography game QuaSim and has been played by over 100 students to learn quantum cryptography basics and cryptography protocols.

Visual distortions play a crucial role in early diagnosis and timely treatment of several eye diseases such as the age-related macular degeneration (AMD). A framework to collect quantitative information about visual distortions, map them to the regions of retina where they may originate, and correlate them to retinal health information obtained using clinical tests, is described in this paper. The resulting composite retinal map can enable physicians to diagnose and treat AMD with minimal manual effort. Our results using the system in practice enabled physicians to study the correlation of distortions to retina lesions with minimal manual overheads.

Spatial pyramid matching (SPM) has achieved impressive successes in analyzing and classifying images across several domains. SPM computes a similarity measure over images by using bag of words similarity score over different levels of coarseness of the images. In this paper we propose a novel, simple approach based on SPM, differential SPM (DSPM) that incorporates finer differences among images while determining image similarity. The approach propagates the differences seen at fine levels to dampen the similarity observed at the coarser levels, thereby highlighting differences among images at small, localized regions. The resulting similarity scores among images can better separate images that match at coarse levels, but have subtle differences. DSPM integrated with K-nearest neighbor classification approaches was used to identify and analyze retinal Optical Coherence Tomography (OCT) images containing normal retinal scans as well as those from subjects with AMD (age-related macular degeneration) and DME (diabetic macular edema). The proposed approach achieved higher classification accuracy with smaller training overheads in comparison to SPM in all cases in our experiments.

Retrieving relevant test cases is a recurring theme in software validation. We present an approach for cluster-based retrieval of test cases for software validation. The approach uses a probabilistic notion of coverage among line-based test profiles and can potentially discover groups of test cases executing a small number of unique lines. The distribution of lines across test profiles are analyzed to determine the number of clusters and generate a clustering structure without any additional user input. We also propose a novel and simple approach to identify test cases that are affected by software changes based on test profiles. It is shown that the clustering structures generated can be used to select affected tests economically to produce high quality regression test suites. The approach is applied to four unix utility programs from a popular testing benchmark. Our results show that the generated number of clusters and their average sizes closely track their estimates based on test profiles. The retrieval of affected tests using the clustering structure is economical and produces a good quality regression test suite.

Samples from clusters of tests are used to automatically predict if tests are to be re-clustered for a modified program. Tests likely to be affected by line level changes are included in the samples by analyzing their execution profiles based on spatial locality. Initial experiments show that the approach can potentially avoid re-clustering in many cases.

We present a new model-based test generation approach using an extended symbolic grammar, which is used as a formal notation for enumerating test cases for communication and reactive systems. Our model-based test generation approach takes inputs a reactive system model, in Live Sequence Charts (LSCs), and a general symbolic grammar serving as preliminary test coverage criteria, performs an automatic simulation for consistency testing on the LSC model specification, and eventually generates an evolved symbolic grammar with relined test coverage criteria. The evolved symbolic grammar can either be used to generate practical test cases for software testing, or be further relined by applying our model-based test generation approach again with additional test coverage criteria.

A formal approach to automatically building a regression test suite whose tests are guaranteed to exercise a given set of system changes is proposed. Given a test and a change, the approach analyzes the test description to provably predict whether or not the test will exercise the change. Adaptive tests whose descriptions involve multiple control paths and support values over commonly used data types are considered. We introduce fully-observable adaptive tests whose descriptions contain all the relevant information about their executions. A structural invariant generated from a test description identifies fully-observable tests and is used to develop a procedure to automatically select tests exercising changes.

Live sequence charts (LSCs) have been proposed as an inter-object scenario-based specification and visual programming language for reactive systems. In this paper, we introduce a logic-based framework to check the consistency of an LSC specification. An LSC simulator has been implemented in logic programming, utilizing a memoized depth-first search strategy, to show how a reactive system in LSCs would response to a set of external event sequences. A formal notation is defined to specify external event sequences, extending the regular expression with a parallel operator and a testing control. The parallel operator allows interleaved parallel external events to be tested in LSCs simultaneously; while the testing control provides users to a new approach to specify and test certain temporal properties (e.g., CTL formula) in a form of LSC. Our framework further provides either a state transition graph or a failure trace to justify the consistency checking results.

A formal approach to select tests for regression testing of changes performed in a system evolution step is proposed. Systems are modeled as extended finite state machines (EFSMs) supporting several commonly used data types including Booleans, numbers, arrays, queues and records. Tests are described using a sequence of input and expected output messages with concrete parameter values. Changes add/delete/replace one or more EFSM transitions. Transitions potentially executed by a test are automatically identified from its description. A simple structural invariant for a test description based on these transitions is introduced. It is shown that for a test description satisfying the invariant it can be accurately determined if a given change affects the test. Affected tests are selected for regression testing of the change. Failure of a description to meet the invariant is analyzed to identify non-observable regions in the description, which are then further analyzed using other system transitions to identify affected tests. We also describe a novel approach based on substitutability of tests to reduce the size of a regression test suite without affecting coverage. The effectiveness of the proposed approach is illustrated by applying it to several examples. Our experiments based on a well-known cost model for regression testing show that the proposed approach is economical for selective re-testing in these examples.

Live sequence charts (LSCs) have been proposed as an inter-object scenario-based specification and visual programming language. In this paper, we introduce a high level computational semantics of LSCs, in the form of a PLAY-tree, to show how a running LSC affects the system behaviors in response to a set of external events. Given a nonempty regular language of external events,the consistency of an LSC specification is defined as whether there exists a corresponding PLAY-tree with all success branches; in case of inconsistency, failure traces can be obtained through failure branches of the PLAY-tree. We also present an algorithm using a memoized depth-first search strategy and an implementation framework in logic programming for consistency checking of LSCs.

Temporal segmentation partitions time series data with the intent of producing more homogeneous segments. It is a technique used to preprocess data so that subsequent time series analysis on individual segments can detect trends that may not be evident when performing time series analysis on the entire dataset.

This paper describes a formal change impact analysis approach for systematic evolution of communicating systems. Systems are modeled using a network of communicating extended finite state machines (CEFSMs) with variables ranging over commonly used data types including numbers, Booleans, arrays, and object fields. Parameterized messages exchanged over queues and shared variables are used for communication. Changes to the system are performed at the transition level by adding/deleting transitions. Given a changed transition, the impacted system transitions are automatically computed using a bounded, selective, state exploration based on the inductive assertion approach. A theorem prover extended with queue axioms is used to discharge the verification conditions. Multiple symbolic values for each variable present in a system state are represented as a set of rewrite rules to minimize state space overheads. Rewrite-rule based procedures are described for reducing the number of symbolic values in system states. We also describe heuristics to identify simultaneously enabled and disabling transitions and describe a procedure to reduce the number of verification conditions generated during the impact analysis. The effectiveness of the proposed approach is illustrated on several applications including Web services and cache coherence protocols.

Abstract Time series segmentation is a promising approach to discover temporal patterns from time‐stamped numeric data. A novel approach to apply time series segmentation to discern temporal information from software version repositories is proposed. Data from such repositories, both numeric and non‐numeric, are represented as item‐set time series data. A dynamic programming algorithm for optimal segmentation is presented. The algorithm automatically produces a compacted item‐set time series that can be analyzed to identify temporal patterns. The effectiveness of the approach is illustrated by analyzing version control repositories of several open‐source projects to identify time‐varying patterns of developer activity. The experimental results show that the segmentation algorithm produces segments that capture meaningful information and is superior to the information content obtained by arbitrarily segmenting software history into regular time intervals. Copyright © 2008 John Wiley & Sons, Ltd.

Time series analysis is a promising approach to discover temporal patterns from time stamped, numeric data. A novel approach to apply time series analysis to discern temporal information from software version repositories is proposed. Version logs containing numeric as well as non-numeric data are represented as an item-set time series. A dynamic programming based algorithm to optimally segment an item-set time series is presented. The algorithm automatically produces a compacted item-set time series that can be analyzed to discern temporal patterns. The effectiveness of the approach is illustrated by applying to the Mozilla data set to study the change frequency and developer activity profiles. The experimental results show that the segmentation algorithm produces segments that capture meaningful information and is superior to the information content obtaining by arbitrarily segmenting time period into regular time intervals.

Abstract A novel, table‐driven approach based on relational database technology is proposed for the design and early validation of cache coherence protocols. A protocol is specified as multiple communicating, multi‐input, multi‐output, finite‐state machines each represented by a relational database table. The tables are automatically generated by solving relational calculus constraints specifying the protocol transactions. Early protocol validation prior to an implementation is performed by testing these tables for several protocol properties expressed using relational queries and database integrity constraints. The debugged tables are automatically mapped to a high‐level hardware implementation while preserving their correctness. The proposed approach has been deployed at Fujitsu Systems Technology Division for the design of their next‐generation multiprocessor and has been highly successful in reducing the overall protocol development time and has discovered several errors early in the design cycle. Copyright © 2006 John Wiley & Sons, Ltd.

Protocols governing communication among the components of a complex system are frequently changed during the design process. To enable faster verification turnaround time, it is important that the existing verification infrastructure continues to be consistent with the changed protocol. In this paper, an approach to identify the effects of protocol changes on runtime monitors is proposed. Runtime monitors are commonly used to observe and verify the dynamic protocol behaviors. Protocols as well as the monitors are modeled using communicating finite state machines. Addition/deletion/replacement of transitions in one or more protocol components may result in similar changes to the monitor transitions. A notion of consistency of a monitor relative to a protocol is introduced. Conditions under which a protocol change necessitates a change to the monitor to preserve relative consistency are identified. Automatic procedures to synthesize new monitors that are guaranteed to be consistent with the changed protocol are described.

While verifying complex protocols, it is often fruitful to consider all protocol contexts in which an interesting set of transitions may appear. The contexts are represented as yet another protocol called observable protocol that may be further analyzed. An efficient approach based on static analysis to compute an over-approximated protocol that includes all the runs of an observable protocol is described. The approach uses dominator relations over state and message dependency graphs. An over-approximation of transitions that occur with an interesting transition in any run are produced, from which a transition relation of the over-approximated protocol is automatically generated. To facilitate systematic state space exploration of the over approximated protocol, it is shown how a series of under-approximations can be generated by identifying parallelism among the transitions using dominators. The effectiveness of the proposed approach is illustrated by model checking several examples including several coherence protocols.