Abstract Online user feedback, like app reviews, can provide valuable insights into software product improvements, offering development teams direct insights into customer experiences, preferences, and pain points. There are many studies that have proposed promising methods to automatically prioritize online user feedback, helping development teams identify the most salient software issues that need to be addressed. However, these methods may not take into account the accessibility-related needs of end users. Our study addresses this limitation by developing a novel approach to analyze and prioritize app store reviews that discuss accessibility concerns. This new approach involves the evaluation of seven distinct machine learning (ML) algorithms, as well as three state-of-the-art large language models (LLMs), all leveraging features of app reviews relevant to accessibility. Utilizing validated accessibility reviews, we assess the effectiveness of our proposed approach and compare its performance with a leading general prioritization tool. The results show that our novel method surpasses the leading general tool in prioritizing accessibility reviews, achieving an F1-score of 83.6%. This represents an improvement over the prior study’s F1-score of 69.0%. Additionally, our approach outperforms the existing method across all three priority classifications, with the most notable improvement seen in the identification of high-priority reviews, where we achieved a + 59.8% increase in F1-score. We hope our findings will inspire more research and innovation in this area and ultimately contribute to a more inclusive and accessible digital landscape for all users.

Folk-wisdom in software engineering suggests that small functions that adhere to the principle of single-responsibility have several advantages over longer, monolithic functions, including improvement in code comprehension. Despite this widespread view, empirical research on the impact of functional decomposition on understanding code is sparse, yet it is central to software development practices.

User feedback on software usage is utilised by developers to improve their software. Software product forums are platforms rich in software-related user feedback, such as forum threads containing bug reports or requests for new features. However, previous studies have mainly focused on analysing user feedback from software product forums as individual sentences, which can lead to missing insights and a lack of understanding of the overall context of forum posts. To fill this gap in research, this work examines user feedback found in software product forum posts to investigate the differences between content classifications found in forum sentences and posts. We manually evaluated software product forum posts collected from two open-sourced software product forums and discovered five new types of user feedback that can only be identified when examining user feedback in the form of forum posts. Additionally, we examined the association between sentence classifications found within software product forums. Our results indicate that contextual information complimenting product improvement insights can be found in software product forums, with a confidence of 0.75 and 0.69 for the association between apparent bug and application usage sentences. This information can be used to reduce manual efforts required to chase up missing contextual information when attempting to understand or fix software issues. We also provide insights into the progression of posts in software product forums at the thread-level, and our progression flowchart can be used to summarise the sequence of events in software product forum threads. Our findings reveal the importance of looking at user feedback within software product forums in the format of forum posts to identify new insights on user feedback for software improvements. Editor’s note: Open Science material was validated by the Journal of Systems and Software Open Science Board.

User feedback is an important resource in modern software development, often containing requirements that help address user concerns and desires for a software product. The feedback in online channels is a recent focus for software engineering researchers, with multiple studies proposing automatic analysis tools. In this work, we investigate the product forums of two large open source software projects. Through a quantitative analysis, we show that forum feedback is often manually linked to related issue tracker entries and product documentation. By linking feedback to their existing documentation, development teams enhance their understanding of known issues, and direct their users to known solutions. We discuss how the links between forum, issue tracker, and product documentation form a requirements ecosystem that has not been identified in the previous literature. We apply state-of-the-art deep-learning to automatically match forum posts with related issue tracker entries. Our approach identifies requirement matches with a mean average precision of 58.9% and hit ratio of 82.2%. Additionally, we apply deep-learning using an innovative clustering technique, achieving promising performance when matching forum posts to related product documentation. We discuss the possible applications of these automated techniques to support the flow of requirements between forum, issue tracker, and product documentation.

Feedback from software users is vital for engineering better software requirements. One tool for extracting requirements from online user feedback is clustering, where the most mentioned topics are found by grouping similar feedback together. For these topics to be understood, clusters have been summarized in previous work using characterizing phrases or sentences. This work evaluates which method of characterization (unigrams, bigrams, trigrams, or sentences) is most effective for understanding the semantic meaning of a whole cluster using feedback from multiple feedback sources. We evaluate multiple characterization methods to determine the ability of each method to create distinct, descriptive characterizations. We further evaluate the amount of requirements relevant characterizations created by each characterization method. We find that unigrams, bigrams, trigrams, and full sentences all perform similarly in distinguishing clusters from each other. However, we find that fewer and more expressive characterizations, such as full sentences, contain more requirements relevant information from a feedback cluster compared to more numerous but less expressive unigrams, meaning a sentence will better summarize the important requirement relevant information from a cluster. Our findings inform the future development of user feedback clustering tools, with different cluster characterization methods being quantitatively measured for the first time.

User feedback on mobile app stores, product forums, and on social media can contain product development insights. There has been a lot of recent research studying this feedback and developing methods to automatically extract requirement-related information. This feedback is generally considered to be the “voice of the users”; however, only a subset of software users provide online feedback. If the demographics of the online feedback givers are not representative of the user base, this introduces the possibility of developing software that does not meet the needs of all users. It is, therefore, important to understand who provides online feedback to ensure the needs of under-represented groups are not being missed.In this work, we directly survey 1040 software users about their feedback habits, software use, and demographic information. Their responses indicate that there are statistically significant differences in who gives feedback on each online channel, with respect to traditional demographics (gender, age, etc). We also identify key differences in what motivates software users to engage with each of the three channels. Our findings provide valuable context for requirements elicited from online feedback and show that considering information from all channels will provide a more comprehensive view of user needs.

The majority of software projects fail, around 71% according to recent research. A shortage of user feedback and missed requirements are cited as primary reasons for failure. There are several prominent online platforms where software users post product feedback, including: app stores, Twitter, issue trackers and product forums. I have identified the study of product forums as a gap in the current requirement mining literature, and have selected them as the focus of this research. Product forums are widely used in the software industry, supporting online discussions between a products users and owners. While their primary function is to help customers use the product, forums are also a rich source of untapped user generated requirements. However, the manual effort to extract these requirements is prohibitively time consuming due to their large volume and inconsistent quality. Analysis tools to assist in requirement mining have been applied successfully to online platforms previously, but as of yet, not in the forum domain, where current techniques may be insufficient. My preliminary research has found that forums contain feedback useful for software maintenance and evolution, including several categories of feedback not identified in the current literature. I have developed forum specific classifiers to help categorise the different feedback in forum posts. I demonstrate that these classifiers significantly outperform a leading app store tool when applied to forums. In this report I present my preliminary findings, then outline my research plan with the final goal of producing an industry evaluated, forum analysis tool.

The modern software landscape is highly competitive. Software companies need to quickly fix reported bugs and release requested new features, or they risk negative reviews and reduced market share. The amount of online user feedback prevents manual analysis. Past research has investigated automated requirement mining techniques on online platforms like App Stores and Twitter, but online product forums have not been studied. In this paper, we show that online product forums are a rich source of user feedback that may be used to elicit product requirements. The information contained in forum questions is different from what has been described in the related work on App Stores or Twitter. Users often provide detailed context to specific problems they encounter with a software product and other users respond with workarounds or to confirm the problem. Through the analysis of two large forums, we identify 18 different types of information (classifications) contained in forums that can be relevant to maintenance and evolution tasks. We show that a state-of-the-art App Store tool is unable to accurately classify forum data, which may be due to the differences in content. Thus, specific techniques are likely needed to mine requirements from product forums. In an exploratory study, we developed classifiers with forum specific features. Promising results are achieved for all classifiers with f-measure scores ranging from 70.3% to 89.8%.