The importance of computational modeling of mobile user interfaces (UIs) is undeniable. However, these require a high-quality UI dataset. Existing datasets are often outdated, collected years ago, and are frequently noisy with mismatches in their visual representation. This presents challenges in modeling UI understanding in the wild. This paper introduces a novel approach to automatically mine UI data from Android apps, leveraging Large Language Models (LLMs) to mimic human-like exploration. To ensure dataset quality, we employ the best practices in UI noise filtering and incorporate human annotation as a final validation step. Our results demonstrate the effectiveness of LLMs-enhanced app exploration in mining more meaningful UIs, resulting in a large dataset MUD of 18k human-annotated UIs from 3.3k apps. We highlight the usefulness of MUD in two common UI modeling tasks: element detection and UI retrieval, showcasing its potential to establish a foundation for future research into high-quality, modern UIs.

Virtual assistants have the potential to play an important role in helping users achieves different tasks. However, these systems face challenges in their real-world usability, characterized by inefficiency and struggles in grasping user intentions. Leveraging recent advances in Large Language Models (LLMs), we introduce GptVoiceTasker, a virtual assistant poised to enhance user experiences and task efficiency on mobile devices. GptVoiceTasker excels at intelligently deciphering user commands and executing relevant device interactions to streamline task completion. For unprecedented tasks, GptVoiceTasker utilises the contextual information and on-screen content to continuously explore and execute the tasks. In addition, the system continually learns from historical user commands to automate subsequent task invocations, further enhancing execution efficiency. From our experiments, GptVoiceTasker achieved 84.5% accuracy in parsing human commands into executable actions and 85.7% accuracy in automating multi-step tasks. In our user study, GptVoiceTasker boosted task efficiency in real-world scenarios by 34.85%, accompanied by positive participant feedback. We made GptVoiceTasker open-source, inviting further research into LLMs utilization for diverse tasks through prompt engineering and leveraging user usage data to improve efficiency.

Mobile apps are ubiquitous in our daily lives for supporting different tasks such as reading and chatting. Despite the availability of many GUI testing tools, app testers still struggle with low testing code coverage due to tools frequently getting stuck in loops or overlooking activities with concealed entries. This results in a significant amount of testing time being spent on redundant and repetitive exploration of a few GUI pages. To address this, we utilize Android’s deep links, which assist in triggering Android intents to lead users to specific pages and introduce a deep link-enhanced exploration method. This approach, integrated into the testing tool Monkey, gives rise to Delm (Deep Link-enhanced Monkey). Delm oversees the dynamic exploration process, guiding the tool out of meaningless testing loops to unexplored GUI pages. We provide a rigorous activity context mock-up approach for triggering existing Android intents to discover more activities with hidden entrances. We conduct experiments to evaluate Delm’s effectiveness on activity context mock-up, activity coverage, method coverage, and crash detection. The findings reveal that Delm can mock up more complex activity contexts and significantly outperform state-of-the-art baselines with 27.2% activity coverage, 21.13% method coverage, and 23.81% crash detection.

Deep Learning (DL) models have rapidly advanced, focusing on achieving high performance through testing model accuracy and robustness. However, it is unclear whether DL projects, as software systems, are tested thoroughly or functionally correct when there is a need to treat and test them like other software systems. Therefore, we empirically study the unit tests in open-source DL projects, analyzing 9,129 projects from GitHub. We find that: (1) unit tested DL projects have positive correlation with the open-source project metrics and have a higher acceptance rate of pull requests; (2) 68% of the sampled DL projects are not unit tested at all; (3) the layer and utilities (utils) of DL models have the most unit tests. Based on these findings and previous research outcomes, we built a mapping taxonomy between unit tests and faults in DL projects. We discuss the implications of our findings for developers and researchers and highlight the need for unit testing in open-source DL projects to ensure their reliability and stability. The study contributes to this community by raising awareness of the importance of unit testing in DL projects and encouraging further research in this area.

Recent years have witnessed a marked increase in both academic proposals and industrial adoptions of blockchain technology. However, a majority of the projects remain at the stage of prototype proposals and their real-world deployment has not met the anticipated level. This gap can be attributed to three major barriers - technical difficulties, human factors, and social context. Most of the existing research leans towards addressing the technical challenges, leaving the human and social aspects inadequately explored. Moreover, a lack of practical insights in the existing blockchain software engineering frameworks further exacerbates the adoption problem. To address these gaps, we introduce a Blockchain-oriented Software Engineering Approach for Higher Adoption Possibility (BOSE-HAP) . This approach emphasizes collaboration, reflective thinking, and iterative development, aiming to bolster implementation consistency and stimulate industry adoption. We have applied this approach in the design, development, and launch of a blockchain credentialing product, CValid.org , in the context of a university-level summer school. The product achieves industry-accepted System Usability Score and has seen successful real-world deployment. In addition, this study embed usability considerations throughout the process, involved a total of 112 stakeholders across different development stages, with 25 of them participating in our in-depth interviews and usability testing. Drawing from our firsthand experience and industrial-grounded findings, we deliver eight reflections and propose five best practice suggestions relevant to blockchain adoption. We believe these insights will provide invaluable guidance for both academic researchers and industry practitioners involved in the field of blockchain technology.

Coronavirus disease 2019 (COVID-19) is an infectious disease caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2). Since its first being reported in December 2019, COVID-19 has spread quickly around the world, and becomes a global pandemic. Previous information sources have a number of problems when providing COVID-19 information web services. First, the information from government or traditional media (i.e., TV and newspaper) is not frequently updated. Second, different layers of the government (state and federal government) may provide contradictory information. Also, there are many rumours spread on social media, which makes it difficult for people to know who and what to trust. Finally, the current information about COVID-19 is fragmented. It takes effort for people to aggregate the information they need to see from different places.

The ongoing COVID-19 pandemic highlights the importance of dashboards for providing critical real-time information. In order to enable people to obtain information in time and to understand complex statistical data, many developers have designed and implemented public-oriented COVID-19 “information dashboards” during the pandemic. However, development often takes a long time and developers are not clear about many people’s information needs, resulting in gaps between information needs and supplies. According to our empirical study and observations with popular developed COVID-19 dashboards, this seriously impedes information acquirement. Our study compares people’s needs on Twitter with existing information suppliers. We determine that despite the COVID-19 information that is currently on existing dashboards, people are also interested in the relationship between COVID-19 and other viruses, the origin of COVID-19, vaccine development, fake new about COVID-19, impact on women, impact on school/university, and impact on business. Most of these have not yet been well addressed. We also summarise the visualization and interaction patterns commonly applied in dashboards, finding key patterns between data and visualization as well as visualization and interaction. Our findings can help developers to better optimize their dashboard to meet people’s needs and make improvements to future crisis management dashboard development.

Developers use different technologies for many software development tasks. However, when faced with several technologies with comparable functionalities, it is not easy to select the most appropriate one, as trial and error comparisons among such technologies are time-consuming. Instead, developers can resort to expert articles, read official documents or ask questions in Q&A sites. However, it still remains difficult to get a comprehensive comparison as online information is often fragmented or contradictory. To overcome these limitations, we propose theDiffTechsystem that exploits crowdsourced discussions from Stack Overflow, and assists technology comparison with an informative summary of different aspects. We first build a large database of comparable technologies in software engineering by mining tags in Stack Overflow. We then locate comparative sentences about comparable technologies with natural language processing methods. We further mine prominent comparison aspects by clustering similar comparative sentences and representing each cluster with its keywords and aggregate the overall opinion towards the comparable technologies. Our evaluation demonstrates both the accuracy and usefulness of our model, and we have implemented our approach as a practical website for public use.

Developers can use different technologies for different software development tasks in their work. However, when faced with several technologies with comparable functionalities, it can be challenging for developers to select the most appropriate one, as trial and error comparisons among such technologies are time-consuming. Instead, developers resort to expert articles, read official documents or ask questions in Q&A sites for technology comparison. However, it is still very opportunistic whether they will get a comprehensive comparison, as online information is often fragmented, contradictory and biased. To overcome these limitations, we propose the DiffTech system that exploits the crowd sourced discussions from Stack Overflow, and assists technology comparison with an informative summary of different comparison aspects. We found 19,118 comparative sentences from 2,410 pairs of comparable technologies. We released our DiffTech website for public use. Our website attracts over 1800 users and we also receive some positive comments on social media. A walkthrough video of the tool demo: https://www.youtube.com/watch?v=ixX41DXRNsI Website link: https://difftech.herokuapp.com/