We introduce FathomGPT, an open source system for the interactive investigation of ocean science data via a natural language interface. FathomGPT was developed in close collaboration with marine scientists to enable researchers to explore and analyze the FathomNet image database. FathomGPT provides a custom information retrieval pipeline that leverages OpenAI’s large language models to enable: the creation of complex queries to retrieve images, taxonomic information, and scientific measurements; mapping common names and morphological features to scientific names; generating interactive charts on demand; and searching by image or specified patterns within an image. In designing FathomGPT, particular emphasis was placed on enhancing the user’s experience by facilitating free-form exploration and optimizing response times. We present an architectural overview and implementation details of FathomGPT, along with a series of ablation studies that demonstrate the effectiveness of our approach to name resolution, fine tuning, and prompt modification. We also present usage scenarios of interactive data exploration sessions and document feedback from ocean scientists and machine learning experts.

Current haptic devices are usually designed to provide one type of force feedback; however, most VR scenarios require versatile force feedback, which may require the integration of different devices to provide various types of forces. In addition, besides the main haptic effects caused by the forces, multiple types of oscillation may also commonly accompany them, which are crucial for improving VR realism and immersion. Therefore, we simulate versatile force feedback by rendering the corresponding types of oscillation as the effects caused by those forces. We take inertia and impact forces as examples in this paper, and achieve versatility using the proposed device, OsciHead, on a head-mounted display (HMD), instead of integrating different devices. By controlling elastic bands' elasticity and stored power, OsciHead uses two rotatable oscillators on both sides of the HMD, in order to render various multilevel and multidimensional oscillation feedback in 2D translation and 2D rotation directions on a head. In an exploratory study, we explored different scenarios in which multiple types of oscillation could be simulated by OsciHead. We then observed oscillation level distinguishability in two just-noticeable difference (JND) studies, and evaluated the oscillation type recognition rates in a recognition study. Based on the results, we performed a VR study, which verified that the inertia and impact feedback simulated by OsciHead enhances realism and achieves versatility.