In the fields of medicine and science, the volume of specialized literature has grown exponentially, containing vast multimodal data-text, images, and tables-that is essential for conveying in-depth scientific insights. However, effectively retrieving, processing, and answering high-level, complex queries from this data remains a significant challenge. In this study, we introduce MedQuery, a multimodal medical knowledge query-answering system that integrates query-based literature retrieval with a response generation module capable of reasoning across multimodal data. Our system begins with literature retrieval from PubMed, using keyword extraction and query alignment to improve document accuracy and relevance. Next, the multimodal processing module processes source documents, extracting images, tables, and text, converting all into a unified textual format. This data is then structured into a global graph capturing relationships among document elements, allowing our system to support a more integrated, in-depth understanding of complex medical queries beyond basic fact retrieval. Extensive evaluations across multiple datasets, including PubMedQA, PubMed-Summarization, and our own MedInquiry dataset, demonstrate that MedQuery outperforms traditional methods and existing commercial AI systems, achieving around 90% win rates in answer quality and a 13-36% improvement in accuracy.

In recent years, multimodal large models in the medical field have garnered widespread attention. However, this focus has primarily been on CT and MRI imaging, inadvertently neglecting the needs of economically underdeveloped regions and specific populations, such as pregnant women. These groups are often unable to utilize CT and MRI due to their prohibitive costs and potential harm to the body. Meanwhile, ultrasound, an economically viable and very low side effects medical imaging technique, has been largely overlooked by researchers. This study introduces a high-quality instruction-tuned Large vision Language Assistant for UltraSound (LLAUS), designed to answer questions about medical ultrasound images, aiming to assist clinicians in impoverished areas to improve the provision of healthcare services. To address the challenge of missing high-quality ultrasound data, we propose the Adaptive Caption Enhancement(ACE) and Adaptive Caption Optimization (ACO) strategies and have developed a high-quality instruction-following dataset. Subsequently, we fine-tune a Large Vision-Language Model (LVLM) using a novel Zoom-In method. By training on high-quality instruction-following datas, LLAUS demonstrates exceptional multimodal ultrasound communication capabilities, assisting in querying ultrasound images based on open-ended instructions. On tasks related to question-answering and caption generation for ultrasound images, LLAUS exhibits strong performance.

With the technological advances in creation and capture of 3D spatial data, new emerging applications are being developed. Digital Twins, metaverse and extended reality (XR) based immersive environments can be enriched by leveraging geocoded 3D spatial data. Unlike 2D spatial queries, queries involving 3D immersive environments need to take the query user’s viewpoint into account. Spatio-visual queries return objects that are visible from the user’s perspective.In this paper, we propose enhancing 3D spatio-visual queries with large language models (LLM). These kinds of queries allow a user to interact with the visible objects using a natural language interface. We have implemented a proof-of-concept prototype and conducted preliminary evaluation. Our results demonstrate the potential of truly interactive immersive environments.

Although federated learning improves privacy of training data by exchanging local gradients or parameters rather than raw data, the adversary still can leverage local gradients and parameters to obtain local training data by launching reconstruction and membership inference attacks. To defend against such privacy attacks, many noises perturbed methods (like differential privacy or CountSketch matrix) have been widely designed. However, the strong defence ability and high learning accuracy of these schemes cannot be ensured at the same time, which will impede the wide application of FL in practice (especially for medical or financial institutions that require both high accuracy and strong privacy guarantee). To overcome this issue, we propose an efficient model perturbation method for federated learning to defend against reconstruction and membership inference attacks launched by curious clients. On the one hand, similar to the differential privacy, our method also selects random numbers as perturbed noises added to the global model parameters, and thus it is very efficient and easy to be integrated in practice. Meanwhile, the random selected noises are positive real numbers and the corresponding value can be arbitrarily large, and thus the strong defence ability can be ensured. On the other hand, unlike differential privacy or other perturbation methods that cannot eliminate added noises, our method allows the server to recover the true aggregated gradients by eliminating the added noises. Therefore, our method does not hinder learning accuracy at all. Extensive experiments demonstrate that for both regression and classification tasks, our method achieves the same accuracy as non-private approaches and outperforms the state-of-the-art defence schemes. Besides, the defence ability of our method against reconstruction and membership inference attack is significantly better than the state-of-the-art related defence schemes.

A corpus of recent work has revealed that the learned index can improve query performance while reducing the storage overhead. It potentially offers an opportunity to address the spatial query processing challenges caused by the surge in location-based services. Although several learned indexes have been proposed to process spatial data, the main idea behind these approaches is to utilize the existing one-dimensional learned models, which requires either converting the spatial data into one-dimensional data or applying the learned model on individual dimensions separately. As a result, these approaches cannot fully utilize or take advantage of the information regarding the spatial distribution of the original spatial data. To this end, in this paper, we exploit it by using the spatial (multi-dimensional) interpolation function as the learned model, which can be directly employed on the spatial data. Specifically, we design an efficient SPatial inteRpolation functIon based Grid index (SPRIG) to process the range and kNN queries. Detailed experiments are conducted on real-world datasets. The results indicate that, compared to the traditional spatial indexes, our proposed learned index can significantly improve the index building and query processing performance with less storage overhead. Moreover, in the best case, our index achieves up to an order of magnitude better performance than ZM-index in range queries and is about 2.7 × , 3 × , and 9 × faster than the multi-dimensional learned index Flood in terms of index building, range queries, and kNN queries, respectively.