A formidable challenge in scheduling user applications lies in collecting and representing the user's goals and requirements. We introduce a “science goal” as a mechanism for users to define scientific objectives and conditions of interest. To provide an abstraction to run applications on an ensemble of edge computing nodes, we implement a two-layered scheduler—cloud and edge scheduler. In this scheduling model, the users submit their goals to the cloud scheduler. These goals are conveyed to the appropriate nodes based on a variety of constraints including geographical area, resource availability, node capabilities, and applicability. The edge scheduler, with complete understanding of the current conditions, assumes the responsibility for executing the applications on the nodes so that the users' science goals are met. This paper provides a framework for the two-layered scheduling model for goal-driven edge computing and motivates and informs its architecture through a case study.

Scientific literature contains large volumes of complex, unstructured figures that are compound in nature (i.e. composed of multiple images, graphs, and drawings). Separation of these compound figures is critical for information retrieval from these figures. In this paper, we propose a new strategy for compound Figure separation, which decomposes the compound figures into constituent subfigures while preserving the association between the subfigures and their respective caption components. We propose a two-stage framework to address the proposed compound Figure separation problem. In the first stage, the subfigure label detection module detects all subfigure labels. Then, in the subfigure detection module, the detected subfigure labels help to detect the subfigures by optimizing the feature selection process and providing the global layout information as extra features. Extensive experiments are conducted to validate the effectiveness and superiority of the proposed framework, which improves the precision by 9% compared to the benchmark.

Multi-focus microscope (MFM) provides a way to obtain 3D information by simultaneously capturing multiple focal planes. The naive method for MFM reconstruction is to stack the sub-images with alignment. However, the resolution in the z-axis in this method is limited by the number of acquired focal planes. In this work we build on a recent reconstruction algorithm for MFM, using information from multiple frames to improve the reconstruction quality. We propose two multiple-frame MFM image reconstruction algorithms: batch and recursive approaches. In the batch approach, we take multiple MFM frames and jointly estimate the 3D image and the motion for each frame. In the recursive approach, we utilize the reconstructed image from the previous frame. Experimental results show that the proposed algorithms produce a sequence of 3D object reconstruction with high quality that enable reconstruction of dynamic extended objects.

We present a primal-dual interior point method (IPM) with a novel preconditioner to solve the ℓ1-norm regularized least square problem for nonnegative sparse signal reconstruction. IPM is a second-order method that uses both gradient and Hessian information to compute effective search directions and achieve super-linear convergence rates. It therefore requires many fewer iterations than first-order methods such as iterative shrinkage/thresholding algorithms (ISTA) that only achieve sub-linear convergence rates. However, each iteration of IPM is more expensive than in ISTA because it needs to evaluate an inverse of a Hessian matrix to compute the Newton direction. We propose to approximate each Hessian matrix by a diagonal matrix plus a rank-one matrix. This approximation matrix is easily invertible using the Sherman-Morrison formula, and is used as a novel preconditioner in a preconditioned conjugate gradient method to compute a truncated Newton direction. We demonstrate the efficiency of our algorithm in compressive 3D volumetric image reconstruction. Numerical experiments show favorable results of our method in comparison with previous interior point based and iterative shrinkage/thresholding based algorithms.

We present a Bayesian approach for 3D image reconstruction of an extended object imaged with multi-focus microscopy (MFM). MFM simultaneously captures multiple sub-images of different focal planes to provide 3D information of the sample. The naive method to reconstruct the object is to stack the sub-images along the z-axis, but the result suffers from poor resolution in the z-axis. The maximum a posteriori framework provides a way to reconstruct a 3D image according to its observation model and prior knowledge. It jointly estimates the 3D image and the model parameters. Experimental results with synthetic and real experimental data show that it enables the high-quality 3D reconstruction of an extended object from MFM.

We present a computer vision system that can transcribe the text on tiny printed labels stacked beneath pinned insects (as found in museum collections). The approach uses multiple views of each label because the labels are often occluded by the pin, the insect specimen, and other labels. Our approach handles occlusion and the extreme viewing angles required to image the stacked labels. Automated image analysis identifies the lines of text and then aligns and rectifies the images. Combining the aligned and rectified images from multiple viewpoints enables us to create a composite image that can be read using optical character recognition tools (OCR) to extract the text. We provide experimental demonstration using both museum specimens and experimental test labels.

This paper presents the design and prototyping of hardware and software to address the problem of rapid and reliable 3D digitization of very large collections of pinned insects. Using the collection at the Field Museum of Natural History (FMNH) as a use case, a pipeline to ingest the entire collection of 4.5 million specimens in circa 1-2 years imposes a few second limit on average processing time per specimen. We describe the design and implementation of multi-camera systems capable of rapidly capturing light field imagery for 3D reconstruction of label surfaces and specimen in single snapshots consistent with this time constraint. With imagery captured using these prototype multi-cameras we demonstrate methods under development for 3D reconstruction of pinned insect specimens and for processing text on label surfaces.

A key trend facing extreme-scale computational science is the widening gap between computational and I/O rates, and the challenge that follows is how to best gain insight from simulation data when it is increasingly impractical to save it to persistent storage for subsequent visual exploration and analysis. One approach to this challenge is centered around the idea of in situ processing, where visualization and analysis processing is performed while data is still resident in memory. This paper examines several key design and performance issues related to the idea of in situ processing at extreme scale on modern platforms: scalability, overhead, performance measurement and analysis, comparison and contrast with a traditional post hoc approach, and interfacing with simulation codes. We illustrate these principles in practice with studies, conducted on large-scale HPC platforms, that include a miniapplication and multiple science application codes, one of which demonstrates in situ methods in use at greater than 1M-way concurrency.

Localization of steerable catheters in minimally invasive surgery is critical with respect to patient safety, surgeon manipulation, and procedural efficacy. While there are many potential benefits to patients including shorter recovery times, less tissue trauma, and lower infection rates than traditional surgeries, localization of surgical tools is still an area of much research. Current technology offers several sensory modalities. However, each system has drawbacks which do not provide a clear best practice. This research focuses on incorporating redundant commonplace surgical sensing technologies to reduce the likelihood of errors, failures, or inherent sensor characteristics causing harm to the patient and/or surgeon while providing accurate localization. Dual particle filters are implemented using both a fluoroscopic-like stereo imaging system and an electromagnetic pose sensor for measurement updates in a prototype catheter testbed. A previously developed catheter model is modified to increase accuracy in the particle filter outputs which are combined using a weighted average based on each filter's particle statistics. Experimental results implementing the combined particle filter multi-modality algorithm in feedback control validates the algorithms ability to provide accurate localization in a surgical setting while overcoming sensor limitations and possible failure modes.

This work demonstrates how a high throughput robotic machine vision systems can quantify seedling development with high spatial and temporal resolution.The throughput that the system provides is high enough to match the needs of functional genomics research. Analyzing images of plant seedlings growing and responding to stimuli is a proven approach to finding the effects of an affected gene. However, with 10 4 genes in a typical plant genome, comprehensive studies will require high throughput methodologies. To increase throughput without sacrificing spatial or temporal resolution, a 3 axis robotic gantry system utilizing visual servoing was developed. The gantry consists of direct drive linear servo motors that can move the cameras at a speed of 1 m/s with an accuracy of 1 μm, and a repeatability of 0.1 μm. Perpendicular to the optical axis of the cameras was a 1 m 2 sample fixture holds 36 Petri plates in which 144 Arabidopsis thaliana seedlings (4 per Petri plate) grew vertically along the surface of an agar gel. A probabilistic image analysis algorithm was used to locate the root of seedlings and a normalized gray scale variance measure was used to achieve focus by servoing along the optical axis. Rotation of the sample holder induced a gravitropic bending response in the roots, which are approximately 45 μm wide and several millimeter in length. The custom hardware and software described here accurately quantified the gravitropic responses of the seedlings in parallel at approximately 3 min intervals over an 8-h period. Here we present an overview of our system and describe some of the necessary capabilities and challenges to automating plant phenotype studies.

Continuum manipulators are gaining widespread acceptance in commercial robotics, particularly in the medical field, where their compliance allows a large benefit for patient safety. However, this compliance also makes precise position control of these manipulators quite difficult. This paper presents two closed-loop control implementations applied to a small scale continuum manipulator. These implementations are both based on manipulator tip position feedback from an electromagnetic sensor. The command tracking and disturbance rejection properties of the two control implementations are shown to be approximately equivalent, and provide improved position control when compared to open-loop control, without sacrificing system stability.

Robotic catheters are gaining widespread use in the medical community for cardiac, neurological and other surgical interventions. However, many of the catheters used in these procedures exhibit non-linear behavior, and thus present many difficulties in implementing effective open-loop control. Systems such as this have been shown to benefit from closed-loop control, however very little investigation has been done into 3D closed loop control of this class of manipulators. Initial investigations by the authors have shown greatly improved positioning accuracy and response with closed-loop control based on feedback from an electromagnetic localization sensor. This paper describes the control approach and experimental results, and provides a robotic catheter system overview.

In recent years, flexible robotic catheter systems have been developed for minimally invasive cardiac surgery. These systems commonly include a passive flexible catheter, which is biocompatible, and an active servo system which provides the actuation for the flexible catheter. To relate control actuation motion of the servo system to catheter motion, models of continuous catheters have been developed. A typical approach is to assume a kinematic model in which the catheter maintains constant curvature along its length when articulated. However, this approach cannot explain the nonlinear behavior of the catheter when the effect of internal friction is considered. In this paper we present a lumped-parameter modeling approach that allows for the inclusion of nonlinear effects, including friction. The proposed approach has been used to model a prototype catheter where internal friction is modeled using a modified Dahl friction model. To evaluate the efficacy of the modeling approach, the simulation results were compared to an experimental catheter prototype. Results show that the proposed approach significantly improves simulation accuracy as compared to the case where friction is not considered. In addition, the effects of friction on catheter performance were investigated using the developed modeling approach.

Traditional robot controllers are not designed to produce human-like reactive motion - movements lasting tens of sample periods and requiring large accelerations. One of the major obstacles to producing reactive motions with contemporary controllers is that they rely on kinematic commands. The performance of short duration motions requiring large accelerations is dominated by the motion's dynamics; kinematic commands without an accurate dynamic model of the robot and task will lead to poor performance. Conversely, this paper presents a biologically inspired “torque command” that allows the dynamics of the motion to be communicated to the controller. The commands can be produced with only minor modifications to any existing control scheme that will not impact traditional operation. In addition, sensory input can be mapped directly to presented commands for latency sensitive tasks. The ability of the new commands to express a broad range of motions with a small number of parameters is shown experimentally. The experimental results also show that the presented torque commands can be used to learn a ball intercept task.

The orientation correlation function is a measure of the spatial range over which nanoscale structures maintain their structural (orientational) similarity. In this paper we describe an image processing system that is used to estimate this correlation function from electron microscope images of the chemically patterned nanoscale structures. We describe the estimation of a robust orientation field from the image and the subsequent estimation of the correlation function from the orientation field. We present results that have been obtained using our image metrology system. Sensitivity of the estimated values with respect to the image processing parameters is also presented.

The ability to control large deflection robotic manipulators without a priori dynamic or kinematic mathematical models is desirable on both the macro and micro scales because of the complexity involved in modeling highly flexible manipulators. Neural network control of a particular large deflection manipulator is detailed. The neural network maps the relationship between the motor position and the pixel location of the large deflection, planar, flexible robot manipulator. The neural network is trained using data extracted from images. The network is tested and shown to have small error relative to the range of motion of the finger. The results indicate that visual servoing techniques can be successfully used to train a neural network model for intelligent control of highly flexible manipulators.

Acquisition, analysis, and classification of repetitive human motion for the assessment of postural stress is of central importance to ergonomics practitioners. We present a two-threshold, multidimensional segmentation algorithm to automatically decompose a complex motion into a sequence of simple linear dynamic models. No a priori assumptions were made about the number of models that comprise the full motion or about the duration of the task cycle. A compact motion representation is obtained for each segment using parameters of a damped harmonic dynamic model. Event classification was performed using cluster analysis with the model parameters as input. Experiments demonstrate the technique on complex motion.

Automated measurement and analysis of human motion during performance of workplace tasks are desirable for ergonomic studies. While numerous technologies exist for accurate measurement of biomechanical data, their use is often not feasible in the workplace environment. We present a digital-video based system suitable for measuring human motion of repetitive workplace tasks. Due to practical considerations, a single-camera solution is exploited by adding some control over the environment. We present an analysis of experiments demonstrating the accuracy of our system.

Automated measurement, analysis, and comparison of human motion during performance of workplace tasks or exercise therapy are core competencies required to realize many telemedicine applications. Ergonomic studies and telemonitoring of patients performing rehabilitation exercises are examples of applications that would benefit from a representation of complex human motion in a form amenable to comparison. We present a representation of joint motion suitable for the analysis of multidimensional angular joint motion time series data. Complex motion is reduced to a concatenation motion segments, where simple dynamic models approximate the observed motion on each segment. This compact representation still enables measurement of statistics familiar to ergonomics practitioners such as cycle length and task duration. An algorithm to obtain this representation from observed motion data (time series) is given. We introduce a metric, based on a kinetic energy-like measure, to compare motions. Experiments are presented to demonstrate the representation, its relationship to previous measures and the applicability of the kinetic energy metric for motion comparison.

Motion capture can be an effective method of creating realistic human motion for animation. Unfortunately, the quality demands for animation place challenging demands on a capture system. To date, capture solutions that meet these demands have required specialized hardware that is invasive and expensive. Computer vision could make animation data much easier to obtain. Unfortunately, current techniques fall short of the demands of animation applications. In this paper, we will explore why the demands of animation lead to a particularly difficult challenge for capture techniques. We present a constraint-based methodology for reconstructing the 3D motion given image observations, and use this as a tool for understanding the problem. Synthetic experiments confirm that these situations would arise in practice. The experiments show how even simple visual tracking information can be used to create human motion but even with perfect tracking, incorrect reconstructions are not only possible but inevitable.

We present a calibration procedure to determine the kinematic parameters of an active stereo system in a robot-centric frame of reference. Our goal was to obtain a solution of sufficient accuracy that the kinematic model information can be used to estimate scene structure given the measured motion/position of the eye and target object's image location. We formulate our problem using canonical coordinates of the rotation group. which enables a particularly simple closed form solution. Additionally, this formulation and solution provides quantitative measures of the resulting solutions. Experiments verify that the solutions are accurate, 3D structure can be estimated from the kinematic models, and the algorithm can indicate when image errors are large enough to produce unreliable results.

In this paper we provide a novel linear algorithm to estimate the direction of translation and the normal to the scene plane, given a stereo pair with known correspondences between the corners in the images. Planar parallax which arises naturally in planar scenes (and can also be defined with respect to virtual planes), is exploited to achieve this. Based on the parallax vectors and the homography induced by the scene plane we calculate the direction of translation and the normal to the scene plane. We also calculate the relative heights of points on the object with respect to the scene plane, given the height of the camera above the plane. The accuracy of the estimated normal with respect to noise in input data is also analyzed empirically.

In order to successfully perform visual servoing of robot manipulators the control algorithm must account for the physical limitations of the manipulator. These constraints define the robot workspace boundary. Any visual servoing control algorithm must avoid this boundary if stability of motion is to be ensured. In the paper, a method is developed by which a desired object path can be interpolated between two arbitrary object poses based on image feature extraction. This path is defined to be a continuous change in object pose between the initial and final desired poses which avoids the workspace boundary. The development of this method involves the parameterization of the 2-D displacement transformation or homography. By decomposing this homography a set of object path poses can be interpolated in either the image plane or reconstructed in the 3-D workspace. Implementation of a visual servoing procedure confirms the validity of the interpolated path with respect to workspace boundary avoidance.

Accurate tracking can facilitate the automatic extraction of metric information from video analysis. Many tracking systems rely on a sufficiently accurate dynamic model. These dynamic models must be either known a priori or learnt. This paper addresses the problem of determining dynamical system models from observed visual motion where it is assumed that the motion cannot be modeled by a single dynamical system. The changes in motion (from one system to another) need to be detected. Previous work has dealt with maintaining multiple hypotheses. For repetitive motion, rather than maintaining multiple hypotheses, one can learn the dynamic models that apply and identify the changes between the models. Specifically, a method for high dimensional motion segmentation is presented. By using a two-step recursive least square algorithm, break points of system dynamics, at which a model switching must be performed are predicted. After segmentation, system identification techniques can be used to fit dynamic models.

Visual-servoing tasks for mobile robots characteristically require the processing of vast amounts of navigational information which can impede the performance of even the most well designed systems. This paper investigates the idea of using a simple human-based strategy to guide a mobile robot to intercept a projectile. This vision based control strategy relies on a single image-based parameter whose sign corresponds to a decision whether to run forward or backward for successful interception. Assuming the ball can be tracked easily in real time by the robot's vision system, motion decision making and control computations are minimized. We examine and develop several motion control strategies which incorporate the robot's velocity and acceleration in addition to this human-based strategy. The simplest strategy is tested on a mobile robot and demonstrates results similar to a human.

The limitations of rigid fingertips in the precise and algorithmic study of manipulation have been discussed in many works, some dating back more than a decade. Despite that fact, much of the work in dexterous manipulation has continued to use the "point-contact" model for finger-object interactions. In fact, most of the existing tactile sensing technologies are not adaptable to deformable fingertips. We report on experimental results obtained with a deformable tactile sensor whose properties are well-suited to manipulation. The results presented here show that the sensor described provides a rich set of tactile data.

While many computer vision algorithms can be used to determine surface orientation information, most require extensive processing (e.g. texture based method). We demonstrate that under fixation, the ratio of angular disparity measures and knowledge of eye position can be used to obtain an estimate of surface orientation. A sensitivity analysis based on accuracy of eye position control parameters is presented. We show that accurate slant can be obtained from a binocular vision system on a mobile robot.

In biological systems a central pattern generator is thought to generate motion patterns for muscular movement that exhibits strong nonlinearity. After learning process, motion is then executed with minimal interaction of the cerebellum, hence freeing up communication channels. Inspired by the biological pattern generator, this paper proposes a scheme to control motor networks with slow communication channels. In the proposed control scheme, motion patterns consist of simple torque pulses. Each joint autonomously generates input torque patterns and the patterns are refined through a searching process. After the searching process, motors can be controlled in an open loop fashion, with minimal management by a central processor. The feasibility of the proposed scheme is demonstrated by a robotic finger system with communication-constrained channels.

Currently, the viability of infrared (IR) as an accurate means of measuring distance depends on extensive prior knowledge of the surface. More specifically, the manner in which a surface scatters, reflects, and absorbs infrared energy is needed to interpret the sensor output as a distance measure. In order to use IR in an unknown environment, one must determine the surface properties during robot operation. The paper details a method of determining the properties of a surface, vis-a-vis infrared reflectance, and subsequently calculating the distance to the surface and the relative orientation of the surface using previously acquired sensory data. We present examples of IR distance measures for various surfaces using known environments as well as examples where the prior knowledge edge has been acquired using sonar sensing. Our results demonstrate that infrared sensors can provide accurate range measurements when used in conjunction with other sensing modalities to fit model parameters during robot operation (i.e. in real-time).

In order to take advantage of the top speed of manipulators, vision cannot be tightly integrated into the motion control loop. Past visual servo control systems have performed satisfactorily with this constraint, however it can be shown that the task execution time can be reduced if the vision system is de-coupled from the low level motor control system. For reaching, there is a trade-off between the accuracy of a motion and the time required to execute motion. In studies of human motor control this tradeoff is quantified by Fitts Law, a relationship between the motion time, the target distance, and the target width. These studies suggest that vision is not used tightly within the control loop, i.e. as a sensor that is servo-ed on, but rather vision is used to determine where the reaching target is and whether target has been reached successfully. Through a simple robotic example we demonstrate that a similar trade off exists between motion accuracy and the motion execution time for visual guided robot reaching motions.

This paper presents an approach to the specification of requirements, and verification of design, for a robot or other intelligent system. The approach is demonstrated on a typical robotic task-visual grasping. Formal mathematical reasoning is used to show that a design conforms to the system requirements. Typically the requirements define safety and functionality constraints on the system and components. Formal analysis allows the system designer to evaluate the system behavior and verify the system parameters that guarantee safe and robust system performance.

Traffic statistics desired by road engineers and planners, and "traffic warning" systems demand real-time performance which precludes the use of batch processing. We apply recent real-time trading techniques along with scene specific tuning of the dynamics to enable the tracker to accurately predict target location and thus reduce the amount of search and/or image processing required. The benefits of learning dynamics for accurate prediction are speed-our tracker operates at frame rate-and smoothing of vibration. Initial calibration of the projective relationship between the image and ground planes enables metric information to be derived from the image positions and velocities without full camera calibration. Results are presented on real-world traffic scenes showing the tracker to be both fast and robust to vibrations which are inevitable in traffic locations.>

The ability to dynamically control the imaging parameters such as camera position, focus, and aperture is satisfied through the use of special purpose hardware such as a robotic head. This paper presents the design and control aspects of the Harvard Head, a binocular image acquisition system. We present three applications of the head in vision tasks concentrating on the computation of depth from controlled camera motion.

Consideration is given to the management of conflicting demands on the use of the motor units of the robot from the active sensors and manipulatory processes. An operating-system-like facility is introduced for managing the conflicting motor requests produced by multiple active sensing and manipulation tasks. A proposed implementation of this sensory-motor management facility based on the input-output-timed-automata (IOTA) abstraction is introduced. The IOTA abstraction is general enough to allow the enforcement of temporal constraints and to permit the specification and modification of task priorities based on the goals and current state of the robot. The IOTA-based sensory-motor operating system acts as a scheduler for various motor requests submitted by the active sensor and manipulation systems. The sensory-motor system is more general than the subsumption architecture of R. Brooks (1986) and can implement in a straightforward fashion alteration of priorities.>

The authors describe a control system for a binocular image acquisition mechanism, for use in mobile robotic systems, which allows shifts in focus of attenuation to be made in a natural, device-independent manner. The control method is based on the modal control technique proposed by R.W. Brockett (Proc. IEEE Robotics Autom. Conf., 1988). The shifts are accomplished by altering the feedback gains applied to the visual feedback paths in the position and velocity control loops of the binocular camera system. By altering these gains, a feature-selection operation can be performed by which the saliency of a given feature is enhanced, while the saliency of other features is reduced. Two experiments performed with the system to demonstrate modal control of attention are discussed.>

Cambridge, MA A vision system for use in a mobile robot system, or in n fixed multi-tasking industrial robot requires attentive control. Attentive control refers to the process by which the direction of gaze of the visual sensors are determined, nlong with the determination of what processing is required to be applied to the sensed images based on the goals of the robot and the tasks it is performing. This paper describes the implementation of a. mnt,ion control system which allows the attentive control of a binocular vision system. Attentive inputs to the system specify the type of visual feedback that the oculo-motor control system will use. The MDL language developed by Brockett [7] i8 used to communicate between the attentive planner and the motion controller.