Ambiguity is pervasive in real-world questions, yet large language models (LLMs) often respond with confident answers rather than seeking clarification. In this work, we show that question ambiguity is linearly encoded in the internal representations of LLMs and can be both detected and controlled at the neuron level. During the model’s pre-filling stage, we identify that a small number of neurons, as few as one, encode question ambiguity information. Probes trained on these Ambiguity-Encoding Neurons (AENs) achieve strong performance on ambiguity detection and generalize across datasets, outperforming prompting-based and representation-based baselines. Layerwise analysis reveals that AENs emerge from shallow layers, suggesting early encoding of ambiguity signals in the model’s processing pipeline. Finally, we show that through manipulating AENs, we can control LLM’s behavior from direct answering to abstention. Our findings reveal that LLMs form compact internal representations of question ambiguity, enabling interpretable and controllable behavior.

Creation and curation of knowledge graphs at scale can be used to exponentially accelerate the discovery, matching, and analysis of diseases in real-world data. While disease ontologies are useful for annotation, integration, and analysis of biological data, codified disease and procedure categories e.g. SNOMED-CT, ICD10, CPT, etc. rarely capture all of the nuances in a patient condition or, in the case of rare disease, may not even exist. Furthermore, there are multiple disease definitions used in data sources and publications, each having its own structure and hierarchy. Mapping between ontologies, finding disease clusters, and building a representation of the chosen disease area are resource-intensive, often requiring significant human capital. We propose the creation and curation of a patient knowledge graph utilizing large language model extraction techniques. In order to expand in volume and scale, knowledge graphs with generalized language capability allow for data to be extracted using natural language rather than being constrained by the exact terminology or hierarchy of existing ontologies. We develop a method of mapping back to existing ontologies such as MeSH, SNOMED-CT, RxNORM, HPO, etc. to ground the extracted entities to known entities in the medical community.We have access to one of the largest ambulatory care EHR databases in the country. To demonstrate the effectiveness of our method, we benchmark our extraction in a test set with over 33.6M unique patients, in the area of patient search. In this case study, we perform a patient search for a rare disease: Dravet syndrome. Dravet syndrome was codified as an ICD10 recognizable disease in October 2020. In the following research, we describe our method of the construction of patient-specific knowledge graphs and subsequent searches for patients who exhibit symptoms of a particular disease. Using patients with confirmed ICD10 codes for Dravet syndrome as our ground truth, we utilize our LLM-based entity extraction techniques and formalize an algorithmic way of characterizing patients in a grounded ontology to assist in mapping patients to specific diseases. Finally, we present the results of a real-world discovery method on Beta-propeller protein-associated neurodegeneration (BPAN), identifying patients with a rare disease, where no ground truth currently exists.

Research has shown that neurons within the brain are selective to certain stimuli. For example, the fusiform face area (FFA) region is known by neuroscientists to selectively activate when people see faces over non-face objects. While the exact mechanisms by which the primary visual system directs information to the correct higher levels of the brain are currently unknown, there are high-level neural mechanisms of perception that we can incorporate in a novel computational model - ones that utilizes lateral and top down feedback in the form of hierarchical competition. We demonstrate that these neural mechanisms provide the foundation of a novel classification framework that rivals traditional supervised learning in computer vision. Additionally, we show that the innate priors built into our architecture support out of distribution generalization on the application of face detection.

Shape analysis tasks, including mesh classification, segmentation, and retrieval demonstrate symmetries in Euclidean space and should be invariant to geometric transformations such as rotation and translation. However, existing methods in mesh analysis often rely on extensive data augmentation and more complex analysis models to handle 3D rotations. Despite these efforts, rotation invariance is not guaranteed, which can significantly reduce accuracy when test samples undergo arbitrary rotations, because the analysis method struggles to generalize to the unknown orientations of the test samples. To address these challenges, our work presents a novel approach that employs graph neural networks (GNNs) to analyze mesh-structured data. Our proposed GNN layer, aggregation function, and local pooling layer are equivariant to the rotation, reflection and translation of 3D shapes, making them suitable building blocks for our proposed rotation-invariant network for the classification of mesh models. Therefore, our proposed approach does not need rotation augmentation, and we can maintain accuracy even when test samples undergo arbitrary rotations. Extensive experiments on various datasets demonstrate that our methods achieve state-of-the-art performance.

Point-of-Care Ultrasound (POCUS) refers to clinician-performed and interpreted ultrasonography at the patient's bedside. Interpreting these images requires a high level of expertise, which may not be available during emergencies. In this paper, we support POCUS by developing classifiers that can aid medical professionals by diagnosing whether or not a patient has pneumothorax. We decomposed the task into multiple steps, using YOLOv4 to extract relevant regions of the video and a 3D sparse coding model to represent video features. Given the difficulty in acquiring positive training videos, we trained a small-data classifier with a maximum of 15 positive and 32 negative examples. To counteract this limitation, we leveraged subject matter expert (SME) knowledge to limit the hypothesis space, thus reducing the cost of data collection. We present results using two lung ultrasound datasets and demonstrate that our model is capable of achieving performance on par with SMEs in pneumothorax identification. We then developed an iOS application that runs our full system in less than 4 seconds on an iPad Pro, and less than 8 seconds on an iPhone 13 Pro, labeling key regions in the lung sonogram to provide interpretable diagnoses.

In this cohort study, we analyzed data collected from Drexel University students, faculty, and staff (age 18 – 79) using machine learning models to gain insight into the significance and predictive capabilities of the the features collected. Data from 126,983 SARS-CoV-2 tests was collected from 16,914 unique individuals from March 4, 2020 to April 24, 2022. Associated symptom data (551,257 reports) was collected through the Drexel University Health Checker App powered by the industry partner, Respond Health. 4,457 people had a positive SARS-CoV-2 test result within the study timeframe. 2,074 of the 4,457 (46.53%) positive cases were in individuals that were fully vaccinated. Given the comprehensive data collected over the entire period of the pandemic, we are able to explore the trends and importance of features to their predictive capabilities. In our experiments and results, we analyze the relative importance of the collected features during different time periods of the COVID-19 evolution and present the trends over time.

The SINDy algorithm has been successfully used to identify the governing equations of dynamical systems from time series data. In this paper, we argue that this makes SINDy a potentially useful tool for causal discovery and that existing tools for causal discovery can be used to dramatically improve the performance of SINDy as tool for robust sparse modeling and system identification. We then demonstrate empirically that augmenting the SINDy algorithm with tools from causal discovery can provides engineers with a tool for learning causally robust governing equations.

In recent years, deep neural networks have had great success in machine learning and pattern recognition. Architecture size for a neural network contributes significantly to the success of neural networks. In this study, we optimize the selection process by investigating different search algorithms to find a neural network architecture size that yields the highest accuracy. We apply binary search on a very well-defined binary classification network search space and compare the results to those of linear search. We also propose how to relax some of the assumptions regarding the data set so that our solution can be generalized to any binary classification problem. We report a 100-fold running time improvement over the naive linear search when we apply the binary search method to our data sets in order to find the best architecture candidate. By finding the optimal architecture for any binary classification problem quickly, we hope that our research contributes to discovering intelligent algorithms for optimizing architecture selection in machine learning.

While deep learning continues to permeate through all fields of signal processing and machine learning, a critical exploit in these frameworks exists and remains unsolved. These exploits, or adversarial examples, are a type of signal attack that can change the output class of a classifier by perturbing the stimulus signal by an imperceptible amount. The attack takes advantage of statistical irregularities within the training data, where the added perturbations can move the image across deep learning decision boundaries. What is even more alarming is the transferability of these attacks to different deep learning models and architectures. This means a successful attack on one model has adversarial effects on other, unrelated models. In a general sense, adversarial attack through perturbations is not a machine learning vulnerability. Human and biological vision can also be fooled by various methods, i.e. mixing high and low frequency images together, by altering semantically related signals, or by sufficiently distorting the input signal. However, the amount and magnitude of such a distortion required to alter biological perception is at a much larger scale. In this work, we explored this gap through the lens of biology and neuroscience in order to understand the robustness exhibited in human perception. Our experiments show that by leveraging sparsity and modeling the biological mechanisms at a cellular level, we are able to mitigate the effect of adversarial alterations to the signal that have no perceptible meaning. Furthermore, we present and illustrate the effects of top-down functional processes that contribute to the inherent immunity in human perception in the context of exploiting these properties to make a more robust machine vision system.

We present a multimodal deep learning framework that can generate summarization text supporting the main idea of an information graphic for presentation to a person who is blind or visually impaired. The framework utilizes the visual, textual, positional, and size characteristics extracted from the image to create the summary. Different and complimentary neural architectures are optimized for each task using crowdsourced training data. From our quantitative experiments and results, we explain the reasoning behind our framework and show the effectiveness of our models. Our qualitative results showcase text generated from our framework and show that Mechanical Turk participants favor them to other automatic and human generated summarizations. We describe the design and results of an experiment to evaluate the utility of our system for people who have visual impairments in the context of understanding Twitter Tweets containing line graphs.

Information graphics, e.g. line or bar graphs, are often displayed in documents and popular media to support an intended message, but for a growing number of people, they are missing the point. The World Health Organization estimates that the number of people with vision impairment could triple in the next thirty years due to population growth and aging. If a graphic is not described, explained in the text, or missing alt tags and other metadata (as is often the case in popular media), the intended message is lost or not adequately conveyed. In this work, we describe a multimodal deep learning approach that supports the communication of the intended message. The multimodal model uses both the pixel data and text data in a single neural network to classify the information graphic into an intention category that has previously been validated as useful for people who are blind or who are visually impaired. Furthermore, we collect a new dataset of information graphics and present qualitative and quantitative results that show our multimodal model exceeds the performance of any one modality alone, and even surpasses the capabilities of the average human annotator.

Deep feed-forward convolutional neural networks (CNNs) have become ubiquitous in virtually all machine learning and computer vision challenges; however, advancements in CNNs have arguably reached an engineering saturation point where incremental novelty results in minor performance gains. Although there is evidence that object classification has reached human levels on narrowly defined tasks, for general applications, the biological visual system is far superior to that of any computer. Research reveals there are numerous missing components in feed-forward deep neural networks that are critical in mammalian vision. The brain does not work solely in a feed-forward fashion, but rather all of the neurons are in competition with each other; neurons are integrating information in a bottom up and top down fashion and incorporating expectation and feedback in the modeling process. Furthermore, our visual cortex is working in tandem with our parietal lobe, integrating sensory information from various modalities.