Interacting with the world is a multi-sensory experience: achieving effective general-purpose interaction requires making use of all available modalities - including vision, touch, and audio - to fill in gaps from partial observation. For example, when vision is occluded reaching into a bag, a robot should rely on its senses of touch and sound. However, state-of-the-art generalist robot policies are typically trained on large datasets to predict robot actions solely from visual and proprioceptive observations. In this work, we propose FuSe, a novel approach that enables finetuning visuomotor generalist policies on heterogeneous sensor modalities for which large datasets are not readily available by leveraging natural language as a common cross-modal grounding. We combine a multimodal contrastive loss with a sensory-grounded language generation loss to encode high-level semantics. In the context of robot manipulation, we show that FuSe enables performing challenging tasks that require reasoning jointly over modalities such as vision, touch, and sound in a zero-shot setting, such as multimodal prompting, compositional cross-modal prompting, and descriptions of objects it interacts with. We show that the same recipe is applicable to widely different generalist policies, including both diffusion-based generalist policies and large vision-language-action (VLA) models. Extensive experiments in the real world show that FuSe is able to increase success rates by over 20% compared to all considered baselines.

This paper reports on methods and results of an applied research project by a team consisting of SAIC and four universities to develop, integrate, and evaluate new approaches to detect the weak signals characteristic of insider threats on organizations' information systems. Our system combines structural and semantic information from a real corporate database of monitored activity on their users' computers to detect independently developed red team inserts of malicious insider activities. We have developed and applied multiple algorithms for anomaly detection based on suspected scenarios of malicious insider behavior, indicators of unusual activities, high-dimensional statistical patterns, temporal sequences, and normal graph evolution. Algorithms and representations for dynamic graph processing provide the ability to scale as needed for enterprise-level deployments on real-time data streams. We have also developed a visual language for specifying combinations of features, baselines, peer groups, time periods, and algorithms to detect anomalies suggestive of instances of insider threat behavior. We defined over 100 data features in seven categories based on approximately 5.5 million actions per day from approximately 5,500 users. We have achieved area under the ROC curve values of up to 0.979 and lift values of 65 on the top 50 user-days identified on two months of real data.

Marine radars have been employed to gather data in applications that require near-continuous monitoring and tracking of objects over a wide area from a single viewpoint, independent of weather and light conditions. However, little attention has been paid toward utilizing such systems for the study of long-term phenomena and detecting anomalous environmental events or hazards that may occur infrequently but have potentially significant impacts on coastal populations. In this paper, we concentrate on tracking features in seasonally ice-covered Arctic coastal ocean environments. We have developed tools for automated analysis of ground-based radar images of landfast ice and moving sea ice to extract ice-floe trajectories and velocity fields, delineate the boundary of stable landfast ice, detect events relevant to coastal populations and identify surface vessels. We employ dense and feature-based optical flow approaches to compute motion fields from the images, active contours for delineation of stable landfast ice, and Hidden Markov Models for machine learning based event detection. We present results from the analysis of sample images jointly with a quantitative evaluation of algorithm performance relative to operator-based assessments.

OBJECTIVES: To determine whether a rule-based algorithm applied to an outpatient electronic medical record (EMR) can identify patients who are pregnant and prescribed medications proved to cause birth defects. DESIGN: A descriptive study using the University of Pennsylvania Health System outpatient EMR to simulate a prospective algorithm to identify exposures during pregnancy to category X medications, soon enough to intervene and potentially prevent the exposure. A subsequent post-hoc algorithm was also tested, working backwards from pregnancy endpoints, to search for possible exposures that should have been detected. MEASUREMENTS: Category X medications prescribed to pregnant patients. RESULTS: The alert simulation identified 2201 pregnancies with 16,969 pregnancy months (excluding abortions and ectopic pregnancies). Of these, 30 appeared to have an order for a non-hormone category X medication during pregnancy. However, none of the 30 'exposed pregnancies' were confirmed as true exposures in medical records review. The post-hoc algorithm identified 5841 pregnancies with 64 exposed pregnancies in 52,569 risk months, only one of which was a confirmed case. CONCLUSIONS: Category X medications may indeed be used in pregnancy, although rarely. However, most patients identified by the algorithm as exposed in pregnancy were not truly exposed. Therefore, implementing an electronic warning without evaluation would have inconvenienced prescribers, possibly hurting some patients (leading to non-use of needed drugs), with no benefit. These data demonstrate that computerized physician order entry interventions should be selected and evaluated carefully even before their use, using alert simulations such as that performed here, rather than just taken off the shelf and accepted as credible without formal evaluation.

We describe a framework for implementing computational thinking in a broad variety of general education courses. The framework is designed to be used by faculty without formal training in information technology in order to understand and integrate computational thinking into their own general education courses. The framework includes examples of computational thinking in a variety of general education courses, as well as sample in-class activities, assignments, and other assessments for the courses. The examples in the different courses are related and differentiated using categories taken from Denning Great Principles of Computing, so that similar types of computational thinking appearing in different contexts are brought together. This aids understanding of the computational thinking found in the courses and provides a template for future work on new course materials. Specific examples of computational thinking in the design category are provided in the context of three distinct courses.