With the rapid advancement of technology, smart healthcare has made significant progress, particularly in medical visual question answering (MedVQA). However, current MedVQA primarily relies on text, whereas practical applications often involve spoken interactions, such as in medical consultations and mobile-based queries. To bridge this gap, we propose a novel task, medical visual spoken-question answering (MedVSA), extending the conventional medical image and text-based question-answering paradigm to include spoken interactions for enhanced applicability. We expand upon four commonly used MedVQA datasets, namely VQA-RAD, SLAKE, PathVQA, and OVQA, by leveraging Alibaba Cloud speech synthesis technology to convert text questions into spoken questions. Various strategies are incorporated to ensure diverse and realistic speech synthesis. The resulting dataset comprises images, corresponding text, and synthesized speech data. Subsequently, we design a single-stage model and a two-stage model to tackle the MedVSA task. For the single-stage model, we directly input the speech and images into our designed whisper self-distillation model to obtain the results. For the two-stage model, we first use the Whisper model to convert the speech into text, then input the converted text and medical images into the self-distillation model to obtain the results. We provide two solutions for the MedVSA task and establish two baselines. Experimental results show that the two-stage model significantly outperforms the single-stage model, indicating that text conversion is crucial for solving MedVSA. This study advances smart healthcare developments by proposing MedVSA and designing two baselines tailored to its specificities. Source code and MedVSA dataset are available at https://github.com/Alivelei/MedVSA.

Optical character recognition (OCR) has been widely studied in previous work. Except for the models used, the recognition accuracy depends most on the resolution of the image to be recognized. To enhance OCR performance, this paper proposes an approach based on a generative adversarial network to improve text image resolution. Our approach uses a perceptual loss function that consists of an adversarial loss, a content loss and an L1 loss. The adversarial loss and the L1 loss are used to ensure the generated super-resolved images are closer to the ground truth high-resolution images. Meanwhile, the content loss is used to ensure the generated super-resolved images and the input low-resolution images have similar features on the basis of perceptual instead of pixel similarity. To evaluate the proposed approach, we compare the recognition accuracies before and after improving the resolution of both English and Chinese text images. The results show that the recognition accuracies on the super-resolved text images obtained with our approach are significantly higher than those on the low-resolution images without processing.

For keyword spotting of historical Mongolian document images, when user provides different instance image for the same query keyword, the performance will vary a lot. This paper proposed an approach to solving the above problem. Particularly, the whole procedure of keyword spotting is divided into two stages. The main task of the first stage is to generate multiple ranking lists for a query keyword. And the aim of the second stage is to merge the multiple ranking lists to form a final ranking. In the first stage, the ranking list of one query keyword is firstly returned by traditional image matching and then a number of instances for the query keyword are obtained using pseudo relevant feedback. Next, each instance of the query keyword can return the corresponding ranking list separately. In the second stage, the multiple ranking lists from the multiple instances of the query keyword are combined by the data fusion technique. The final ranking will be taken as the retrieval results of the query keyword. The experimental results show that the proposed approach can significantly improve the performance of keyword spotting for the historical Mongolian document images.

There are many classical Mongolian historical documents which are reserved in image form, and as a result it is difficult for us to explore and retrieve them. In this paper, we investigate the peculiarities of classical Mongolian documents and propose an approach to recognize the words in them. We design an algorithm to segment the Mongolian words into several Glyph Units(Glyph Unit abbr. GU). Each GU is consisted of no more than three characters. Then we used a three-stage method to recognize the GUs. At the first stage, all the GUs are classified into nine groups by decision tree using three features of the GUs. At the second stage, the GUs in each group are classified individually by five independent BP Neutral Networks whose inputs are other five feature vectors of the GUs. At the last stage, the five results of each GU group from the above five classifiers are combined to provide the final recognized result. The recognition rate of the Mongolian words in our experiment achieves 71%, indicating that our method is effective.